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CHAPTER  I 


SOURCE  SELECTION 

The  Source  Selection  Decision  Problem 

Air  Force  system  acquisition  projects  involve 
contracts  at  stages  throughout  the  project  life  for  the 
procurement  of  services  and  equipment  toward  fulfilling  the 
system  acquisition  objectives.  Typically,  each  procure¬ 
ment  involves  the  solicitation  of  offers  followed  by  an 
evaluation  process  which  produces  information  from  which  a 
choice  is  made  to  determine  the  contract  award. 

Source  Selection  Decision-Making 

The  evaluation  leading  to  the  award  decision 
(source  selection)  is  the  composite  product  of  the  results 
of  independent  expert  assessments  of  a  variety  of  aspects 
of  the  offers  under  consideration.  While  these  aspects 
may  cover  a  wide  range  of  criteria,  they  can  be  conveniently 
grouped  into  five  major  areas:  (l)  technical,  (2)  opera¬ 
tional,  (3)  logistics,  (4)  management,  and  (5)  cost.  Each 
area  may  in  turn  be  broken  out  into  more  specific  items 
which  themselves  may  shred  out  into  more  discrete  segments 
called  "factors"  ( 1 7: pp • 3-2, 3-3) •  The  regulatory  documents 
which  are  reviewed  in  this  study  give  broad  guidance  on 


the  use  of  subjective  and  objective  methods  of  integrating 
expert  assessments  of  the  separate  aspects  of  offers  into 
consolidated  evaluation  reports  to  provide  a  basis  for  the 
source  selection  decision.  The  evaluation  should  provide 
a  balanced  appraisal  of  all  significant  factors  with  a 
high  level  of  quality  and  consistency  to  facilitate  an 
objective,  impartial,  equitable,  and  economic  comparative 
analysis  of  competing  offers. 

Determining  an  integrated  evaluation  of  an  offer 
from  factor  assessments  involves  two  processes:  (l)  scor¬ 
ing  and  (2)  weighting. 

(1)  Scoring  is  the  allocation  of  a  comparative 
"value"  to  each  factor  being  assessed.  The 
value  may  be  expressed  by  allocating  a 
numerical  score,  by  color  coding,  by  ranking, 
by  narrative,  or  by  a  combination  of  these 
methods . 

(2)  Weighting  is  the  process  of  giving  to  the 
score  value  of  each  factor  a  coefficient 
which  reflects  the  relative  importance  of 
the  factor  in  the  final  evaluation.  In 
practice,  weighting  may  be  done  by  object¬ 
ively  allocating  numerical  coefficients  or  by 
subjective  comparisons. 
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As  the  foregoing  suggests,  a  number  of  empirical 


models  have  been  developed  as  a  guide  in  making  evaluations 
for  source  selection  decision-making.  Because  of  the  nec¬ 
essarily  large  number  of  participants  in  factor  assessment 
and  the  variety  of  integrating  techniques  available,  it  is 
difficult  to  justify  that  current  procedures  provide  the 
required  objectives  of  quality  and  consistency  in  Air  Force 
source  selections. 

Decision-Making  Problems 

A  recent  study  ( 10:49)  of  source  selection  proc¬ 
edures  in  Aeronautical  Systems  Division  (ASD)  showed  that 
different  groups  of  source  selection  evaluators  rated  the 
same  proposals  differently.  An  examination  (8:119)  of 
numerical  scoring  and  weighting  schemes  concluded  that 
small  relative  changes  in  item  weights  and  item  scores 
can  overturn  the  order  of  evaluations  when  differences  bet¬ 
ween  scores  are  small  fractions  of  the  scores.  Awareness 
of  shortcomings  in  numerical  rating  schemes  has  led  to  a 
"strong  trend  toward  rating  proposal  elements  using  a 
combination  narrative  and  color  coding  system  ^T:pp.9, 10_7." 
While  this  trend  avoids  the  specific  criticisms  of  numer¬ 
ical  rating  schemes,  there  is  no  hard  evidence  to  show 
that  more  subjective  methods  than  numerical  weighting  come 
closer  to  obtaining  consistent  and  accurate  source  selec¬ 
tion  decisions. 
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The  Research  Need 


There  remains  a  need  for  a  source  selection  evalua 
tion  procedure  which  is  capable  of  giving  demonstrably 
consistent  results  with  different  assessment  groups. 

Source  selection  from  among  complex  competing  offers  is  a 
multi-attribute  decision-making  situation. 

Some  recent  academic  discussions  of  possible 
applications  of  multi-attribute  decision  theory  to  milit¬ 
ary  logistics  problems  are  discussed  in  the  literature 
review.  It  is  believed  that  theory  developments  in  this 
field  offer  a  base  from  which  to  examine  an  actual  source 
selection  process  to  determine  a  source  selection  decision 
maker's  visibility  of  the  assessment  factors  in  relation 
to  the  source  selection  criteria.  A  sufficiently  rigor¬ 
ous  examination  may  provide  new  insights  into  source 
selection  that  will  enable  the  development  of  more  effect¬ 
ive  management  of  the  process. 


Literature  Review 


Policy  and  Procedural  Background 

Department  of  Defense  Directive  (DODD)  4105.62 
(20:2)  provides  source  selection  policy  and  procedures  for 
the  acquisition  of  major  defense  systems,  and  states  three 
primary  objectives  to  be  met  as  a  result  of  the  source 
selection  process. 


4 


The  prime  objectives  of  the  process  are  to 
(a)  select  the  source  whose  proposal  has  the  highest 
degree  of  realism  and  credibility  and  whose  perform¬ 
ance  is  expected  to  best  meet  Government  objectives 
at  an  affordable  cost;  (b)  assure  impartial,  equitable, 
and  comprehensive  evaluation  of  competitors'  proposals 
and  related  capabilities;  and  (c)  maximize  efficiency 
and  minimize  complexity  of  solicitation,  evaluation 
and  the  selection  decision. 

The  major  systems  acquisition  process  is  a  complex 
one.  It  consists  of  a  sequence  of  specified  phases  of 
program  activity  and  decision  events  directed  to  the 
achievement  of  program  objectives  in  the  acquisition  of 
defense  systems.  Each  major  weapon  system  acquisition 
program  has  its  unique  features,  and  therefore,  no  two 
programs  are  identical.  If  one  were  to  compare  various 
programs,  a  number  of  differences  would  immediately  surface 
to  include  differences  in  time,  cost,  technology,  manage¬ 
ment,  and  contracting  approach.  Despite  the  differences, 
however,  the  basic  acquisition  process  is  common  to  all 
programs.  As  such,  all  programs  are  driven  through  the 
process  toward  a  common  goal  of  obtaining  for  the  Govern¬ 
ment  "the  most  advantageous  contract--prices,  quality  and 
other  factors  considered  /J8  :p.  1  -302. 2j.  " 

DODD  4105.62  (20:2)  provides  guidance  to  achieve 
this  objective: 

Each  DOD  Component  shall  develop,  and  consistently 
apply,  procedures  which  create  the  environment  for  an 
impartial,  balanced  and  realistic  appraisal  of  all 
proposals  submitted. 
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Air  Force  Regulation  (AFR)  70-1 5 »  Source  Selection 
Policy  and  Procedures.  (l'J'sp.1  —  1  )  establishes  policy, 
assigns  authority  and  responsibilities,  and  prescribes 
implementing  procedures  for  source  selection.  It  also 
states  the  main  objective  of  the  source  selection  process: 

The  prime  objective  of  proposal  evaluation  and 
source  selection  is  to  assure  impartial,  equitable, 
and  comprehensive  evaluation  of  competitive  proposals 
and  to  assure  selection  of  that  source  whose  proposal, 
as  submitted,  offers  optimum  satisfaction  of  the 
Government's  objectives  including  cost,  schedule,  and 
perf ormanc e . 

A  typical  source  selection  process  is  composed  of 
a  structured  organization  which  consists  of  a  Source 
Selection  Authority  (SSA),  Source  Selection  Advisory 
Council  (SS AC),  and  a  Source  Selection  Evaluation  Board 
(SSEB)  (l7Jp*1-*0.  The  source  selection  process  itself  is 
initiated  as  a  result  of  the  submission  and  approval  of  a 
Source  Selection  Plan.  This  plan,  the  key  planning  docu¬ 
ment  for  the  conduct  of  the  source  selection  process,  is 
normally  prepared  by  the  project  officer  charged  with 
effecting  the  procurement  of  the  system  (l7sp»2-l).  The 
Source  Selection  Plan  usually  includes,  among  other  things, 
"basic  evaluation  criteria  to  provide  a  basis  for  the  more 
detailed  shredout  by  the  SSAC  and  SSEB  for  use  in  the 
solicitation,"  a  description  of  the  SSEB  evaluation  and 
rating  methodology  and  the  SSAC  analysis  technique,  and  a 
schedule  of  events,  identifying  and  listing  the  source 
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selection  activities  within  a  time  framework  (l7spp.2-1, 

2-2). 

The  Contract  Def initization  Group  (CDG)  is  a  part 
of  the  SSEB  organization.  Its  role  is  to  negotiate  def¬ 
initive  contracts  with  all  offerors  determined  to  be  in 
the  competitive  range.  The  CDG  manages  all  communications 
with  the  offerors  and  is  advised  by  a  Cost  Panel.  The 
primary  purpose  of  the  Cost  Panel  is  to  provide  an  evalua¬ 
tion  of  the  most  probable  cost  to  the  Government  of  each 
offeror's  proposal  (lsl6).  While  technical  and  cost 
evaluations  by  different  evaluators  are  held  simultaneously , 
they  are  kept  apart  to  prevent  the  technical  evaluation 
from  being  biased  by  cost  considerations. 

After  proposals  are  received,  the  evaluation  period 
commences  with  the  SSEB  examining  and  conducting: 

...  an  in-depth  review  of  the  relative  merits 
of  each  proposal  against  the  requirements  in  the 
solicitation  document  and  the  evaluation  criteria 
established  by  the  SSAC .  The  evaluation  function 
must  be  thoroughly  conducted,  objective,  fair,  and 
economical  :p.  1  -6j, 

A  summary  report  of  findings  by  the  SSEB  is  then 
prepared  and  submitted  to  the  SSAC.  This  SSEB  Evaluation 
Report  is  basically  a  summary  of  the  results  obtained 
after  evaluating  each  proposal  against  the  standard  crit¬ 
eria  set  forth  by  the  SSAC  (l7!p.1-6). 


AFR  70-15  ( 1 7 s pp. 1 -5 ♦ 1 -6 )  establishes  the  SSAC ' s 


duties  and  responsibilities.  These  include,  among  others: 

(1)  Establish  the  evaluation  criteria,  using  the 
general  guidance  set  forth  in  the  approved  Source 
Selection  Plan. 

(2)  Establish  the  relative  importance  of  the 
evaluation  criteria  in  a  form  for  use  in  the  solicita¬ 
tion  document. 

(3)  Establish  the  evaluation  criteria  weights 
for  SSAC  use  when  numerical  scoring  techniques  are 
employed . 

(4)  Review  the  findings  of  the  SSEB  and,  when 
numerical  scoring  has  been  used,  apply  the  established 
weights  to  the  evaluation  results. 

(5)  Prepare  the  SSAC  Analysis  Report  (comparative 
analysis)  based  on  the  SSEB  Evaluation  Report. 

Basically,  this  part  of  the  process  consists  of  a 
review  of  the  SSEB  Evaluation  Report  by  the  SSAC,  after 
which  an  evaluation  of  proposals  is  again  conducted  against 
the  SSAC  criteria.  A  Source  Selection  Advisory  Council 
Analysis  Report  is  then  submitted  to  the  SSA.  This  com¬ 
parative  analysis  report  consists  of  a  "proposal  versus 
proposal"  evaluation  that  should  help  the  SSA  make  an 
objective  selection  decision  ( 1 7 :pp • 1 -1 , 1 -2» 1 -5» 1 -6 ) . 

The  SSA  is  ultimately  responsible  for  the  proper 
conduct  of  the  proposal  evaluation  and  source  selection 
process.  Therefore,  he  should  strive  for  a  source  selec¬ 
tion  process  that  will  provide  him  with  the  information 
necessary  to  make  the  most  objective  selection  decision 
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possible 


The  SSA  must  be  presented  sufficient  indepth 
information  on  each  of  the  competing  offerors  and 
their  proposals  to  make  an  objective  selection 
decision.  The  5SAC  Analysis  Report  and  oral  briefing 
should  be  presented  to  the  SSA  in  a  manner  which 
accomplishes  this  objective.  The  SSAC  presents 
findings  and  analyses  but  does  not  make  recommenda¬ 
tions  to  the  SSA  unless  specifically  requested 

17:p.  1  -2_7  • 

In  the  final  analysis,  the  degree  of  success  that 
the  SSA  will  attain  in  making  an  objective  decision  will 
depend  on  the  extent  to  which  a  logical,  consistent,  and 
systematic  approach  is  established.  AFR  70-15  (  1 7 * P •  —3 ) 
provides  guidance  for  the  establishment  of  evaluation 
criteria  and  rating  systems  to  be  used  in  evaluating 
offerors'  proposals: 

The  specific  evaluation  criteria  must  be  included 
in  the  solicitation  document  and  enumerated  in  terms 
of  relative  order  of  importance  of  those  significant 
factors  which  will  form  the  general  basis  for  proposal 
evaluation  and  selection/contract  award  ...  The 
rating  system  shall  be  structured  to  enable  the  SSA 
to  identify  the  significant  differences,  strengths, 
weaknesses,  and  risks  associated  with  each  proposal 
and  subsequent  definitized  contract  .  .  .  The  rating 
system  may  be  entirely  narrative,  or  may  employ 
numerical  scoring  and  weights  or  a  descriptive  color 
code  in  conjunction  with  narrative  assessments.  The 
important  task  in  either  rating  system  is  the  integ¬ 
rated  assessment  of  all  aspects  of  the  evaluation, 
analysis,  and  negotiation  process. 

AFR  70-15  (l7!p.3“^-)  relies  on  the  evaluator's  own 
judgment  while  performing  an  evaluation: 
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How  an  evaluator  approaches  the  task  of  evaluation 
is  up  to  his  own  judgment  based  on  his  experience.  The 
method  by  which  it  is  accomplished  is  dependent  on  what 
he  feels  best  suits  the  particular  circumstances  ,  .  . 

It  is,  however,  important  that  all  evaluators  be  con¬ 
sistent  in  their  approach  to  evaluation.  Failure  to 
do  so  will  result  in  distortion  of  the  true  value  of 
the  proposals. 


Purpose  of  Source  Selection  Procedures 

The  Logistics  Management  Institute  briefed  the 
Defense  Blue  Ribbon  Panel  on  the  subject  of  defense  proc¬ 
urement  policy  and  weapon  systems  acquisition  in  August 
1969: 

Formal  procedures  were  established  for  selecting 
contractors  for  major  development  or  large  production 
efforts.  These  procedures  required  evaluation  of 
proposals  according  to  pre-established,  point  grading 
criteria  and  a  review  of  the  documented  results  of 
the  grading  system.  The  objective  was  to  reduce  the 
influence  of  subjective  judgments  in  the  selection  of 
contractors  and  to  encourage  objective  evaluation  of 
all  proposals  by  responsible  offerors  ^9:21__/, 

The  essential  decision-making  process  in  source 
selection  involves  weighing  and  judging  complex  issues 
arising  from  the  assessment  of  the  many  factors  which  make 
up  competing  offers.  The  issues  are  evaluated  by  separate 
expert  groups  with  different  perceptions  of  the  ultimate 
acquisition.  Weighting  of  issues  is  subject  to  the  biases 
of  the  weighters.  Overall  policies  may  be  overwhelmed  by 
the  goals  of  the  organizational  subsystems  involved  in  the 
process.  The  source  selection  decision-maker  requires 
information  which: 
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(l)  relates  to  the  acquisition  policy  and 


objectives 

(2)  is  free  from  bias 

(3)  is  equitably  weighted 

(4)  can  withstand  scrutiny  and  be  repeatable 
with  different  assessors. 

Finally,  the  decision-maker  requires  the  informa¬ 
tion  in  a  form  which  is  digestible  and  which  will  assist 
him  to  exercise  judgment  in  the  fullest  possible  knowledge 
of  the  choices  available. 

Theoretical  Background 

A  preliminary  survey  of  general  literature  in 
decision-making  suggests  that  there  are  research  findings 
which  might  be  applied  to  the  experience  of  existing 
empirical  source  selection  models  to  develop  an  improved 
understanding  of  the  source  selection  process. 

Simon  (l6:272)  proposed  the  concept  of  bounded 
rationality  as  a  feature  of  management  decision-making. 

He  reasoned  that  decision-makers  in  complex  situations 
"satisficed"  the  choices  available  to  them.  They  used  only 
that  part  of  the  available  information  which  they  perceived 
to  enable  them  to  make  a  satisfactory  rather  than  an  op¬ 
timal  decision.  The  cautions  and  reservations  expressed 
in  current  Air  Force  and  DOD  source  selection  regulations 

1 1 


confirm  an  awareness  of  the  difficulties  of  source  selec¬ 


tion  decision-making.  Because  of  this  inherent  complexity, 
"satisficing”  continues  to  play  a  significant  part  in 
source  selection  decision-making  as  a  practical  necessity. 

These  views  have  support  in  a  recent  research  which 
utilized  multiple  linear  regression  techniques  to  examine 
source  selection  in  an  Air  Force  procurement  division. 
Milligan  (lO:vi)  attempted  to  determine  whether  or  not  the 
evaluation  criteria  contributed  significantly  to  the  rating 
a  proposal  received  and  how  program  managers  and  super¬ 
visors  make  source  selection  decisions.  He  found  that 
people  do  not  always  use  all  the  information  available  to 
them  in  making  source  selection  decisions: 

Thesis  results  suggest  that  source  selection 
decisions  are  not  similar  across  organizations  within 
the  AF  division.  Furthermore,  subjects  did  not 
utilize  all  the  information  available  to  them  in  making 
decisions.  People  often  chose  to  utilize  only  a  part 
of  the  available  information  in  arriving  at  a  decision. 

Dawes  (3*180-188)  demonstrated  the  "bootstrap" 
effect  of  making  policy  a  conscious  element  of  the  decision 
model.  When  policy  was  defined  or  "captured"  in  the  model, 
decisions  became  more  aligned  to  policy.  A  similar  effect 
in  Air  Force  source  selection  decision-making  was  suggested 
by  Milligan  when  he  showed  that  source  selections  were  more 
consistent  among  experienced  source  selection  staff  when 
given  a  policy  in  the  decision  task  statement  than  when 
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they  were  given  the  task  with  no  formal  policy. 

While  trying  to  improve  the  proposal  evaluation 
phase  of  the  source  selection  process,  Dycus  (5s256) 
conducted  an  Evaluator  Preference  Survey  in  which  he 
attempted  to  measure  the  attitudes  and  evaluative  judgment 
of  a  quasi-sample  of  33  experienced  DOD  technical  proposal 
evaluators.  Although  he  found  that  the  evaluators  had  a 
favourable  attitude  toward  proposal  evaluations,  survey 
responses  indicated  a  need  for  improvement  of  the  evaluative 
procedures : 


.  .  .  survey  data  indicated  considerable  room 
for  the  government  to  improve  proposal  evaluation 
mechanics.  Most  evaluators  indicated  they  reinter¬ 
preted  scored  evaluation  criteria.  There  was  only 
moderate  judgment  that  scores  evaluation  criteria 
and  rating  scales  were  "good"  and  "fair". 

Dycus  recommended  that  experimental  research  be 
conducted  in  order  to  improve  the  proposal  evaluation 
aspect  of  source  selection.  He  further  suggested  that  such 
research  would  improve  evaluation  rating  scales,  evaluation 
criteria  for  scoring,  determine  preferred  evaluation  mech¬ 
anics,  and  improve  scoring  discrimination: 

End  product  of  proposed  experimental  research 
would  be  a  proposal  evaluation  guide  that  defines  a 
preferred  rating  scale,  and  directs  the  evaluators 
in  how  to  make  their  evaluation  scorings.  Such  a 
guide  would  improve  the  quality  and  discrimination  of 
proposal  evaluation  scores,  and  attest  to  the  prac- 
tical  value  of  applied  procurement  research  25 6 
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The  primary  goal  of  source  selection  is  to  arrive 
at  an  objective  selection  decision.  However,  several 
problems  exist  which  limit  the  ability  to  accomplish  this 
goal.  The  work  of  Beard  (2s iv)  in  his  study,  "The  Applica¬ 
tion  of  Multi-Attribute  Utility  Measurement  (MAUM)  to  the 
Weapon  Systems  Source  Selection  Process",  identifies  five 
problem  areas  that  presently  limit  the  ability  to  fully 
accomplish  an  objective  evaluations 

These  problems  are:  current  weapon  systems 
development  is  multidimensional  and  does  not  allow 
for  evaluation  on  a  single  dimension  -  an  array  of 
attributes  must  be  evaluated;  performance  evaluation 
is  in  many  cases  a  subjective  attribute  and  judgment 
can  be  influenced  by  biased  viewpoints;  the  current 
color  coded  evaluation  procedure  provides  results 
that  can  be  washed  out  and  are  arrived  at  wholistic; 
the  current  numerical  evaluation  procedure  provides 
results  that  can  be  very  close,  may  tend  to  level 
out  results  or  obscure  the  more  important  issues; 
and  costs. 

MAUM  is  a  ten-step  procedural  approach  developed 
from  multi- at tribute  utility  theory  by  Dr.  Ward  Edwards 
to  objectively  address  important  decisions  when  selecting 
among  various  alternatives  having  multiple  attributes 
(technical,  logistic,  and  operations  evaluation  factors) 
(2:8).  It  provides  a  framework  for  scoring  and  weighting 
attributes  in  such  a  way  as  to  ensure  significant  discrim¬ 
ination  between  the  scores  allocated  to  substantially 
different  proposals.  Using  this  approach.  Beard  (2:43) 
concluded  that  the  objectivity  required  in  source  selec¬ 
tion  decisions  can  be  attained: 


MAUM's  proceduralized  methodology  greatly  reduces 
the  influence  individual  bias  can  have  in  evaluation 
results.  The  use  of  value  curves  and  the  philosophy 
of  "operationally  defining"  evaluation  factors  will 
result  in  much  more  objective  evaluations.  MAUM’s 
procedures  preclude  inconsistent  application  of 
evaluation  standards  over  time. 

Basically,  Beard  argued  that  since  the  present 
source  selection  evaluation  process  considers  various 
proposals  having  multiple  attributes  (evaluation  items  and 
factors),  MAUM's  ability  to  evaluate  decisions  having  more 
than  one  attribute,  aspect,  criterion,  or  dimension  helps 
eliminate  the  problems  presently  encountered  in  the  proc¬ 
ess  (2:42). 

There  has  been  concern  regarding  the  numerical 
scoring  and  weighting  system  used  to  evaluate  offerors' 
proposals,  specifically,  the  sensitivity  of  total  scores 
to  small  variations  in  the  choices  of  item  weights  and  in 
item  scores.  In  a  paper  presented  to  the  Sixth  Annual 
Procurement  Research  Symposium,  Lee  was  concerned  with  the 
possibility  of  these  variations  causing  "offeror  A  to  have 
a  greater  total  score  than  offeror  B  in  one  case,  while 
making  B's  score  exceed  A's  in  another  (8:123)."  Lee 
concluded  that: 

The  order  of  numerical  scores  of  proposals  can 
be  overturned  by  small  relative  changes  in  item 
weights  and  item  scores  whenever  differences  between 
scores  are  small  fractions  of  the  scores,  even  when 
item  weights  meet  all  the  requirements  of  AFR  70-15 
and  AFLC  Supplement  1  to  that  regulation. 
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Some  Recent  Propositions 

Vaid  (21:12)  has  described  DOD  material  acquisi¬ 
tion  decision-making  as  a  value-building  process  and 
proposed  the  use  of  a  theory  of  analytic  hierarchies  to 
clarify  the  multi-criteria  choice  situation  involved.  A 
key  element  of  the  theory  is  the  use  of  two-dimensional 
comparison  matrices  to  refine  expert  estimates  of  value 
scores  in  terms  of  the  value  structure  of  the  organization. 
A  claimed  advantage  of  the  scheme  was  that  it  is  relatively 
simple  to  construct  and  administer  in  a  complex  organiza¬ 
tion.  He  concluded  that  the  process  drives  toward 
consensus  and  provides  a  truly  wholistic  approach  to 
decisions . 

A  proposed  application  of  multi-criteria  decision 
theory  to  a  specific  Air  Force  acquisition  planning 
decision-making  scenario  by  DeWispelare,  Sage,  and  White 
(4: p.l-15)  identified  two  major  theoretical  application 
techniques:  Multiple  Objective  Optimization  Techniques 

(MOOT)  and  Multiple  Attribute  Utility  Theory  (MAUT).  It 
was  suggested  that  both  MOOT  and  MAUT  are  mental  constructs 
to  approaching  multiple  criteria  decision  situations  and 
that  there  are  practically  no  fundamental  differences 
between  their  analytical  structures.  However,  while  MOOT 
may  more  quickly  identify  non-dominant  solution  sets, 
decision-maker  preference  (weighting)  emerges  more 
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efficiently  through  MAUT.  Dewispelare,  Sage,  and  White 
developed  a  methodology  of  combining  the  organizationally 
desirable  features  of  MOOT  and  MAUT.  The  methodology  has 
been  tested  in  the  Air  Force  and  has  been  demonstrated  to 
be  an  acceptable  and  desirable  approach  to  improving  the 
efficiency  of  decision-making.  The  research  offers  en¬ 
couragement  of  the  practicality  of  developing  the  applica¬ 
tion  of  multi-criteria  decision  theory  to  source  selection 
scoring  and  weighting. 

Practical  Considerations 

The  literature  review  to  this  stage  suggests  that 
the  major  problems  in  achieving  effective  source  selection 
evaluations  include  consistency,  equitable  weighting  of 
factors,  and  policy  visibility.  Recent  research  has 
focussed  on  methodology  for  improving  the  quality  of 
estimates  of  value  score  and  attribute  weighting. 

In  general,  source  selection  practitioners  are 
averse  to  the  use  of  mathematical  models  of  subjective 
judgment  which  use  numerical  scoring  (for  example,  (6:89)). 
There  is  a  feeling  that  numerical  scoring  methods  inhibit 
the  freedom  of  the  decision-maker  to  make,  and  justify, 
subjective  decisions.  During  recent  years,  there  has  been 
a  strong  trend  toward  using  a  combination  narrative  and 
color-coding  system  in  rating  proposal  elements.  When 
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used,  the  following  is  an  example  of  how  the  color-coding 
system  may  be  applied  (l:10): 

Blue  -  Exceeds  specified  performance  or  capab¬ 
ility  and  excess  is  useful,  high 
probability  of  success,  no  significant 
weaknesses . 

Green  -  Average,  meets  most  objectives,  good 
probability  of  success,  deficiencies 
can  be  corrected. 

Yellow  -  Weak,  low  probability  of  success,  signi¬ 
ficant  deficiencies,  but  correctable. 

Red  -  Key  element  fails  to  meet  intent  of 
Request  for  Proposal  (RFP). 

The  source  selection  evaluation  process  provides 
for  a  tendency  to  "wash"  the  evaluation  of  proposals  toward 
an  acceptable  standard.  This  effect  appears  to  be  due  in 
part  to  the  conservativeness  of  evaluators  at  the  lower 
levels.  These  evaluators  appear  to  be  reluctant  to  rate 
a  proposal  as  unacceptable  and  so  eliminate  it  from  the 
competition.  Evaluators  at  these  levels  seem  to  avoid  this 
kind  of  decision,  deferring  it  to  the  higher  level  to  make 
such  a  determination.  The  cumulative  effect  reduces  the 
visibility  at  the  higher  levels  of  the  process  of  the 
overall  worth  of  the  different  proposals  when  compared 
agains  t  s  tandards . 

It  is  believed  that  detailed  study  of  the  value 
building  processes  in  actual  source  selections  is  necessary 
before  significant  conclusions  can  be  made  about  the 
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practical  application  of  multi-attribute  decision-making 
theory  to  improving  source  selection. 
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CHAPTER  II 


RESEARCH  APPROACH 

Research  Objectives 

The  objective  of  this  research  was  to  examine  the 
value  building  processes  in  an  actual  source  selection  case 
and  to  establish  its  correspondence  with  the  theoretical 
constructs  of  multi-attribute  decision  theory. 

Scope  of  Research 

Formal  source  selection  evaluation  procedures  are 
mandatory  only  for  new  development  programs  requiring  $100 
million  or  more  RDT&E  fluids  or  projected  to  require  more 
than  $500  million  production  funds,  or  other  programs 
specifically  designated  (19:2).  However,  the  objectives 
of  source  selection  remain  the  same  in  all  procurements 
regardless  of  dollar  value,  and  responsible  officers  are 
required  to  demonstrate  a  systematic  and  consistent  approach 
to  the  source  selection  decision.  The  problems  of  offer 
evaluation  and  decision-making  are  similar  for  all  pro¬ 
curements  and  vary  only  in  size  and  scope. 

This  research  was  directed  to  a  detailed  study 
of  a  source  selection  process  undertaken  within  the 
standardized  formal  procedural  framework  used  within  the 
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Aeronautical  Systems  Division  (ASD) .  However,  it  is 
considered  that  it  will  provide  a  basis  for  study  of  more 
general  cases  of  government  source  selection. 

The  research  included  a  series  of  interviews  with 
source  selection  practitioners  and  administrators  to 
identify  perceptions  of  the  strengths  and  weaknesses  of 
the  empirical  source  selection  models  in  current  use. 

Research  Question 

The  question  addressed  in  the  study  introduced  by 
this  paper  may  be  summarized  in  the  following  ways 
Can  a  detailed  study  of  an  actual  source 
selection  process  establish  a  relationship 
with  theoretical  multi-attribute  decision¬ 
making  models  which  will  provide  means  for 
improving  the  management  of  source  selection? 

Research  Methodology 


Discussion 

The  process  of  source  selection  may  be  pictured 
conveniently  as  a  value  hierarchy  in  the  manner  described 
by  Waid  (21:14).  In  source  selection,  the  levels  of 
hierarchy  are  typically: 


source 

area 

item 

factor 

sub-factor 
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The  values  of  each  component  which  contribute  to 
the  decision  rise  progressively  through  the  hierarchy, 
successively  being  refined,  until  they  reach  the  source 
decision  level.  At  each  level,  a  series  of  weighted 
combinations  of  individual  component  scores  takes  place 
to  arrive  at  a  new  set  of  scores  to  enter  the  weighting 
process  at  the  next  higher  level  (Figure  l). 

Researchers  have  modeled  this  kind  of  combina¬ 
tional  process  in  many  applications  as  a  linear  multi¬ 
attribute  utility  model  (22:122-124).  The  model  is 
expressed  in  the  form: 

Y  =  BlXl  ♦  B2x2  *  B3x3  +  .  .  .  ♦  Bnxn 

where  Y  is  the  value  (score)  outcome  of  the 
process,  and  xr  is  the  value  of  the  nth 
component,  and  Bn  is  the  weighting  coeffic¬ 
ient  of  the  nth  value. 

The  linear  model  is  consistent  with  the  procedures  outlined 
in  AFR  70-15  (l7*P»2-6)  and  the  source  selection  policy¬ 
capturing  research  of  Milligan  (10:12). 

The  underlying  assumptions  of  the  linear  model  are 
that  the  components  are  (22:123)* 

(1)  independent  -  to  avoid  double  counting,  and 

(2)  unidimensional  -  the  scores  should  be  realist¬ 
ically  seen  as  adding  to  the  decision  dimen¬ 
sion,  and 
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Value  Level 


FIGURE  1  -  Source  Selection  Value  Hierarchy 
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(3)  compensating  -  high  scores  on  some  components 
will  compensate  for  low  scores  on  others,  and 

(4)  relevant  -  the  components  should  be  relevant 
over  all  contexts,  and 

(5)  exhaustive  -  all  appropriate  components  should 
be  included,  but 

(6)  determinant  -  the  components  should  be  impor¬ 
tant  to  the  selection. 

When  comparing  a  number  of  competing  multi-attribute 
options  (proposals)  as  in  source  selection,  meaningful 
comparisons  are  made  when  the  coefficients  B  remain  con¬ 
stant  for  each  calculation  of  Y. 

This  kind  of  model  implies  a  straightforward  way 
of  combining  objective  component  scores  through  a  value 
hierarchy  to  evaluate  competing  proposals.  The  combined 
scores  become  absolute  comparative  values  at  the  source 
level  which  should  provide  a  clear  basis  for  the  source 
selection  decision. 

However,  numerical  scoring  systems  have  been  almost 
universally  rejected  in  ASD  as  a  suitable  means  of  source 
selection  for  anything  but  the  simplest  procurements. 

The  major  objections  to  numerical  scoring  arising 
ou-t  of  practical  experience  are: 
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(1)  variations  between  offers  are  "averaged  out" 

i 

so  that  major  deviations  from  standard 
become  obscured  in  the  final  score. 

(2)  The  allocation  of  objective  scores  narrows 
the  options  of  the  decision-maker  constrain¬ 
ing  him  to  the  highest  numerical  score. 

These  objections  are  supported  in  a  stated  reluct¬ 
ance  of  evaluation  personnel  to  use  the  full  range  of 
numerical  scoring  scales  and  the  findings  by  Lee  (8:123) 
that  when  total  scores  are  close,  small  variations  in 
component  scores  can  overturn  the  result.  The  latter 
gives  concern  that  the  former  can  lead  to  a  scoring  result 
that  is  not  optimal. 

ASD  prefers  the  use  of  more  subjective  color  coding 
scoring  systems  which  are  believed  to  give  the  combiner  at 
each  level  in  the  value  hierarchy  greater  flexibility  in 
the  subconscious  weights  that  he  applies  to  subordinate 
component  color  scores  when  allocating  his  own  color  score 
to  the  whole  of  the  group  of  components  within  his  res¬ 
ponsibility. 

In  most  important  acquisitions,  the  potential 
contractors  are  experienced  and  competent  in  government 
contracting.  They  have  developed  an  intimate  knowledge  of 
the  government's  requirements  during  prior  negotiations 
and  understand  the  source  selection  process.  Their 


proposals  are  therefore  constructed  to  closely  conform 
to  the  expected  criteria.  The  outcome  is  that  all  propo¬ 
sals  tend  to  meet  the  main  acquisition  requirements  and 
that  differences  between  them  are  small.  Differences 
between  proposals  tend  not  to  be  evident  in  the  compara¬ 
tive  color  score  allocated  at  higher  levels  in  the  value 
hierarchy.  The  Source  Selection  Advisory  Council  is  often 
obliged  to  look  below  the  highest  hierarchical  levels  to 
detect  differences  which  may  become  justifiable  bases  on 
which  to  make  accept/reject  decisions.  It  appears  in 
practice  that  at  each  level  of  combination  of  scores  there 
is  an  effect  which  "washes-out"  the  visibility  of  signifi¬ 
cant  factors  which  may  be  a  basis  for  acceptance  or 
rejection  when  considered  with  the  whole. 

Research  Hypothesis 

Even  with,  the  use  by  ASD  of  color-coded  scores  in 
the  value  hierarchy,  the  concept  of  a  linear  combinatorial 
model  remains  valid  if  meaningful  numerical  equivalent 
scores  may  be  given  to  the  color  code.  However,  because 
the  color-coding  technique  of  scoring  does  not  lend  itself 
to  a  priori  allocation  of  objective  attribute  weights 
(other  than  a  simple  ranking  of  order  of  importance),  it 
is  possible  to  bias  the  model  during  application.  The  bias 
effect  may  be  represented  simply  by  extending  the  model  by 
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a  constant  term  B0,  such  that: 

Y  =  Bn  +  B,x,  +  B-X_  +  .  .  .  +  B  x 
0  11  2  2  n  n 

AFR  70-15  suggests  a  suitable  scale  of  numbers  in  the 

range  0-10  to  equate  to  color  scorings.  This  scale  was 

adopted  as  a  basis  to  score  color-code  ratings  used  in 

source  selection: 

blue  -  10 
green  -  6 

yellow  -  2.5 

red  -  0 

The  value-building  processes  in  actual  source 
selection  cases  were  examined  to  see  if  a  fit  could  be 
established  between  the  actual  processes  and  the  extended 
model.  A  survey  of  recent  ASD  source  selections  showed 
that  suitable  cases  could  be  identified  with  sufficient 
proposals  and  components  to  be  able  to  conduct  a  multiple 
regression  analysis  of  the  allocated  cell  value  to  the 
component  values  for  each  cell  of  the  decision  hierarchy. 

Analysis  was  conducted  by  the  multiple  regression 
procedures  in  the  Statistical  Package  for  the  Social 
Sciences  (SPSS),  (12:328),  available  on  the  Cyber  CDC 
6600  Computer.  The  basic  test  hypothesis  was: 

H0  !  B0  =  0 
H1  S  B0  ji  0 

Having  established  the  possible  nature  of  the  value¬ 
building  equation  when  color-scoring  was  used  as  the 
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discriminant,  a  computer  model  was  constructed  to  simulate 
a  series  of  value-building  situations  to  examine  the  rela¬ 
tive  performance  of  color-scoring  and  numerical  scoring 
methods  as  value  discriminants. 

Interviews  were  held  with  experienced  source 
selection  practitioners  in  ASD  to  gain  a  fuller  apprecia¬ 
tion  of  the  empirical  source  selection  process  and  to 
validate  the  model  assumptions. 

The  outcome  enabled  some  conclusions  to  be  made 
about  the  way  the  source  selection  process  functions  and 
its  relationship  to  the  DOD  objectives. 
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CHAPTER  III 


ANALYSIS  AND  MODELING 

Review  and  Analysis  of  Cases 

A  preliminary  survey  of  source  selection  cases 
completed  in  ASD  over  the  period  1975  to  1980  was  made  to 
identify  cases  with  sufficient  historical  data  of  the 
value  building  process  to  facilitate  detailed  analysis 
of  value  building  hierarchies. 

Three  cases  were  identified  as  being  suitable  for 
analysis,  and  permission  was  granted  by  the  Commander,  ASD 
to  examine  the  records  in  detail.  The  selected  cases 
involved  teams  of  4l  ,  43  and  70  evaluators  and  4,  5  and  9 
proposals  respectively.  All  cases  used  combinations  of 
color  and  narrative  scoring  techniques. 

Within  each  case,  it  was  sought  to  isolate  indiv¬ 
idual  value  building  cells  which  met  criteria  for  multiple 
regression  analysis. 

The  criteria  sought  for  analysis  were: 

(l)  The  higher-level  composite  "value"  given  to  a 
value  building  cell  was  expressed  in  compara¬ 
tive  terms  to  the  expression  of  values  of 
the  component  parts  or  attributes  (i.e., 
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colors,  or  by  generic  descriptive  groupings 
such  as  "Exceeds  Standards",  "Meets  Standard", 
"Fails  to  Meet  Standard",  "Unacceptable"). 

(2)  There  were  a  sufficient  number  of  proposals 
in  relation  to  the  number  of  independent 
components  so  that  the  multiple  regression 
synthesis  of  relationship  was  valid,  i.e., 
the  number  of  attributes  (variables)  was  less 
than  the  number  of  proposals  (sample  size) 
(12:329). 

Twelve  value  building  cells  were  identified  which  met  the 
criteria. 


Results  of  Regression  Analysis 


The  twelve  value  cells  examined  had  from  three  to 

thirteen  components.  However,  where  all  proposals  scored 

the  same  for  a  component,  that  component  was  eliminated 

from  inclusion  in  the  equation  as  being  discriminating. 

The  SPSS  multiple  regression  technique  was  then  applied  to 

evaluate  the  relationship  of  implicit  value: 

Y  *  B0  +  B-x,  +  B„x„  +  .  .  .  +  B  x 
u  1  1  2  2  n  n 

Further  components  were  eliminated  in  the  regression 

analysis  because  of  multi-collinearity  or  because  they 

were  below  a  0.01  inclusion  level.  As  a  result,  all  value 

cells  reduced  to  five  or  less  significant  components  in 
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the  analysis.  B0  was  found  to  be  equal  to  zero  with  95$ 
confidence  in  only  two  of  the  twelv'a  relationships  so 
derived.  Values  of  B0  identified  with  95$  confidence  were 
two  in  the  range  0  to  7«7  and  eight  in  the  range  -10.3  to  0. 

The  B^  values  reflect  the  evaluator's  perception 
of  the  relative  importance  of  the  components  of  the 
decision  cell.  The  structure  of  the  source  selection 
process  requires  that  weightings  be  determined  in  advance 
of  and  separately  from  the  component  evaluations  (l7sp.3“7). 
It  is  difficult,  if  not  impossible,  for  the  evaluator  to 
express  absolute  weights  in  numerical  terms  when  using 
color  or  narrative  scoring.  The  observed  practice  is  only 
to  rank  components  in  order  of  relative  importance  to  each 
other  at  the  outset  and  it  is  the  judgment  of  the  evaluator 
which  determines  the  implicit  relative  weight  actually 
accorded  to  each  component  at  the  time  of  final  determina¬ 
tion  of  the  composite  score. 

The  value  of  B0  may  be  perceived  as  a  measure  of 
the  cell  evaluator's  adjustment  of  the  weighted  composite 
score  against  a  subjective  benchmark.  It  reflects  a  sub¬ 
jective  readjustment  of  the  value  of  the  competing  proposals 
in  an  attempt  to  portray  a  relationship  between  them  and 
the  perceived  standard. 

The  composite  scorer  for  the  cell,  therefore, 
undergoes  a  complex  process  of  mental  weighting  and  re- 
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evaluation  of  the  component  score  data  in  arriving  at  a 
value.  When  using  the  color  code/narrative  approach  of 
ASD,  he  is  constrained  to  express  the  judgmental  outcome 
by  one  of  four  discrete  "values”  (red,  yellow,  green  or 
blue),  shaded  as  necessary  by  narrative  support. 

Modeling  the  Value-Building  Process 

Dr.  Lee  has  shown  (8:119)  that  when  numerical 
scoring  schemes  are  used,  the  order  of  numerical  scores 
of  the  whole  are  sensitive  to  small  relative  changes  in 
component  weights  and  scores  whenever  differences  in  scores 
are  small . 

This  part  of  the  research  was  concerned  with  how 
the  sensitivity  of  the  model  was  affected  when  a  four- 
increment  scale  of  scoring  was  used  instead  of  a  relatively 
continuous  numerical  scoring  scale;  and  to  see  what  effect 
the  introduction  of  the  B0  value  adjustment  had  on  the 
discriminating  power  of  the  model. 

In  considering  the  discrimination  between  different 
proposals  when  color-scoring  is  used,  it  was  evident  that 
a  difference  becomes  significant  when  component  scores  are 
near  the  "border-line"  of  an  incremental  range  on  the 
scoring  scale.  Because  of  the  discontinuous  nature  of 
the  discriminating  effect  of  score  differences,  it  was 
decided  that  the  problem  could  be  most  conveniently 


32 


examined  by  means  of  a  computer  simulation.  A  computer 
model  was  therefore  constructed  to  find  out  the  effective¬ 
ness  of  the  scoring  system  used  by  ASD  as  a  means  of 
discriminating  between  offers  of  various  differences,  and 
to  compare  the  performance  of  color  scoring  with  numerical 
scoring. 

Computer  Model 

The  model  was  constructed  to  simulate  a  value 
building  cell  of  five  components,  the  whole  value  of  which 
is  represented  by  Y  where: 

Y  =  B0  +  B^x^  +  ®2X2  +  ^3X3  +  +  B5x5 

The  values,  of  the  components  of  the  cell  were 

randomly  generated  for  five  value  cells,  representative  of 
the  situation  of  evaluating  five  competing  proposals.  The 
data  were  generated  to  represent  five  sets  of  proposals  of 
differing  degrees  of  "goodness"  so  that  the  discriminating 
properties  of  the  model  might  be  observed.  For  each  set 
of  five  lots  of  simulated  data,  the  five  item  values  (Y) 
were  calculated  and  the  highest  scored  proposal  was  det¬ 
ermined.  Multiple  sets  of  data  were  tested  over  a  range 
of  values  of  B0  and  B^  and  goodness  levels  to  determine 
the  frequency  of  selection  of  the  "best"  proposal  for 
each  set  of  independent  variables.  The  model  also  rep¬ 
licated  the  process  using  the  raw  numerical  scores  of  x^ 
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instead  of  the  four- increment  color  scoring  scale 


Assumptions  of  the  Model 

Examination  and  analysis  of  the  case  histories 
suggested  that  the  following  were  reasonable  assumptions 
on  which  to  base  a  model  synthesis: 

(1)  Each  component  item  of  the  value  cell  is 
independent,  i,e.f  no  multi-collinearity 
exists . 

(2)  The  value  attributed  to  each  proposal  in 
the  whole  may  be  conceptualized  on  a  scale 
of  1  to  10  and  that  the  limit  of  perception 
of  objective  difference  of  values  of  pro¬ 
posals  so  conceptualized  is  2  per  cent. 

If  the  objective  difference  is  less  than  2 
per  cent,  then  the  "best"  bid  will  be 
selected  on  subjective  factors. 

(3)  Goodness  has  consistency.  A  proposal  for 
which  the  evaluation  is  "good"  in  an  item 
may  be  expected  to  perform  at  a  "good"  level 
on  the  average  across  all  factors  that  make 
up  the  components  of  the  item  evaluation. 

(4)  Evaluators  tend  to  judge  components  against 
the  standard  on  a  continuum  before  allocating 
discrete  color  or  descriptive  scores. 
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Parameters  in  the  Model 

Goodness  Level 

In  order  to  be  able  to  examine  the  discrimination 
of  the  model,  it  was  necessary  to  simulate  data  represent¬ 
ing  the  evaluated  component  scores  of  proposals  of  differ¬ 
ing  quality  or  "goodness". 

Assumption  3  states  that  the  values  attributed  to 
the  components  of  a  "good"  proposal  in  a  value  cell  will 
cluster  about  a  value  higher  than  the  value  about  which 
bids  of  lesser  goodness  will  cluster.  To  simulate  this 
concept,  "goodness"  levels  were  modeled  on  a  scale  of  1 
to  10.  The  designated  "goodness"  level  was  set  as  the 
mode  of  a  continuous  triangular  frequency  distribution. 

A  computer-generated,  uniformly  distributed  pseudo-random 
number  was  then  put  against  the  cumulative  distribution 
curve  of  the  triangular  distribution  to  derive  a  "goodness” 
number.  The  numbers  (AX)  so  derived  were  then  reduced  to 
a  scale  of  color-equivalent  incremental  values  (x)  as 
listed  in  AFR  70-15  07:p.3-6)  as  follows: 

If  (AX. LT. 1.25)  then  X=0 
If  (AX.GE. 1 . 25. and. AX. LT. 4. 25)  then  X=2.5 
If  ( AX . GE .4.25. and . AX . LT . 8 )  then  X=6 
If  (AX.GE. 8)  then  X=10 
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Sets  of  five  proposals  were  simulated  with  each 


proposal  in  the  set  being  of  a  designated  "goodness"  level. 
Each  proposal  consisted  of  a  value  cell  with  five  compon¬ 
ents  . 

In  each  set  of  five  proposals  the  "best"  proposal 
was  put  at  a  goodness  level  of  10  and  goodness  levels  of 
the  remaining  proposals  put  at  10  per  cent  decrements.  In 
each  successive  goodness  set,  the  interval  between  the 
"best"  and  the  "second  best"  proposal  was  increased  by  10 
per  cent. 

The  resulting  goodness  levels  of  the  proposals  in 
the  sets  are  shown  in  Table  I . 

TABLE  I 

GOODNESS  LEVELS  OF  SIMULATED  PROPOSALS 

Proposal  Number 
1  2  3  4  5 

10  10  9  8  7 

10  9  8  7  6 

10  8  7  6  5 

10  7  6  5  4 

10  6  5  4  3 


Proposal 

Set 

1 

2 

3 

4 


5 


Proposal  sets  were  not  extended  beyond  set  number 
5  because: 

(1)  it  was  judged  that  a  10:6  quality  ratio  was 
representative  of  the  largest  gap  between 
proposals  which  would  merit  formal  source 
selection  procedures,  and 

(2)  the  difference  between  the  "best”  and  "second 
best"  proposal  could  no  longer  be  regarded 

as  "small". 

Weighting  Coefficients  (B^) 

When  total  value  is  determined  by  the  expression 
Y  =  B-jX^  +  B2x2  +  +  .  .  .  +  Bnxn  .  .  .  (1) 

and  Y  and  x^  are  both  scored  on  the  same  value  scale, 
then: 

£  B  =  1 . (2) 

i=1 

Typically  (l3:33)»  weighting  coefficients  when 
used  in  source  selection  are  put  at  values  which  are 
multiples  of  0.1.  Within  these  guidelines,  there  are 
seven  possible  sets  of  values  for  B^  when  five  terms  are 
included  in  the  total  value  expression  (Table  II). 
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TABLE  II 


POSSIBLE  SETS  OF  VALUES  OF  WEIGHTING  COEFFICIENTS 

Set  Number  Value  of  Bj 

1 
2 

3 

4 

5 

6 

7 

All  seven  possible  sets  of  B^  values  were  included  in  the 
computer  model. 


0.6, 

0.1  , 

0.1  , 

0.1  , 

0.1 

0.5, 

0.2, 

0.1  , 

0.1  , 

0.1 

0.4, 

0.3, 

0.1  , 

0.1  , 

0.1 

0.4, 

0.2, 

0.2, 

0.1  , 

0.1 

0.3, 

0.3, 

•* 

CM 

• 

o 

0.1, 

0.1 

0.3, 

0.2, 

0.2, 

0.2, 

0.1 

o 

• 

ro 

«• 

0.2, 

0.2, 

0.2, 

0.2 

Introduction  of  B0  to  the  Model 

The  analysis  of  twelve  value-building  cells  from 
three  source  selection  cases  revealed  B0  values  ranging 
from  -10.3  to  +7.7.  The  number  of  cells  examined  is  a 
small  sample  compared  with  many  source  selection  cases. 
Given  the  small  sample  size,  it  was  not  possible  to  make 
significant  conclusions  about  the  real  limits  of  range 
and  frequency  of  occurrence  of  B0  values  when  color¬ 
scoring  or  narrative-scoring  systems  of  value  expression 
were  used.  However,  it  was  sufficient  for  this  research 
to  observe  the  possibility  of  occurrence  of  significant 
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B0  values.  When  a  real  value  of  B0  was  introduced  into  the 
value  equation  and  all  values  sum  to  1  ,  as  expressed  in 
equation  (2),  it  was  necessary  to  modify  equation  (l)  to 
retain  the  same  scoring  scales  for  Y  and  x^,  so  that: 

Y  =  B0  +  (1-B0/S)(B1X1+B2x2+B3x3  +  .  .  .  +Bnxn)  ...  (3) 

where  S  is  the  scoring  scale  for  Y  and  x^. 

Since  the  concept  of  the  model  was  that  the  values 
of  the  components  (x^)  were  additive  toward  the  value  of 
the  whole,  and  as  BQ  approached  S  the  value  of  Y  approached 
B0,  the  maximum  practical  limit  of  B„  was  S. 

For  the  purposes  of  the  model,  three  values  were 
chosen  for  the  adjustment  parameter: 

B0  =  +7 

Bo  «  0 

B0  =  -7 

as  a  basis  to  observe  the  effects  of  inclusion  of  BQ  in 
the  value  building  equation. 

The  Computer  Program 

The  computer  program  to  simulate  the  operation  of 
the  model  is  listed  at  Appendices  A1-A3. 

The  program  was  arranged  to  give  80  simulations  of 
data  for  each  goodness  set,  providing  2000  simulated  data 
points.  The  data  were  processed  to  find  the  value  or 
score  for  each  bid/goodness  set  combination  and  to  select 
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the  highest  scoring  bid  for  each  simulation.  The  proportion 
of  each  bid  selected  over  the  80  simulations  was  calculated. 
The  outputs  which  the  program  provided  were: 

(1)  A  frequency  table  for  each  goodness  set  of 
per  cent  each  proposal  selected  in  the  first 
run  of  simulations  for  the  five  goodness  sets 
and  three  values  of  B„  and  7  sets  of  B. 
coefficients . 

(2)  A  histogram  for  each  frequency  table. 

(3)  A  summary  table  of  the  frequency  of  selection 
of  the  "best"  proposal  (bid  number  1 )  for 
each  run  of  simulations  against  goodness  set, 

Bc  value  and  B.^  coefficient  set. 

The  program  was  arranged  to  run  the  80  simulations 
five  times,  each  time  from  a  new  random  number  base.  The 
five  runs  were  repeated  using  absolute  numerical  scores 
for  discrimination  instead  of  incremental  color  scoring  to 
provide  a  basis  for  comparison  between  the  two  scoring 
methods . 

The  ten  summary  tables  are  presented  at  Appendices 
B1-B10  which  show,  for  the  same  proposal  data: 

(l)  Frequency  of  selection  of  proposal  number  1 

against  goodness  and  B0  and  B^  sets  for  color 
scoring  for  five  runs. 
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(2)  Frequency  of  selection  of  proposal  number  1 
against  goodness  and  B0  and  B_^  sets  for 
numerical  scoring. 

Analysis  of  Output  of  Computer  Model 

The  computer  model  experiment  was  intended  to  study 
the  effects  on  the  numbers  of  times  the  "best"  proposal 
was  selected  by  successively  varying  the  four  factors: 
weighting  coefficients  (B^),  goodness  (LG),  adjustment 
parameter  (B0),  and  scoring  method. 

The  four  factors  or  treatments  were  varied  in  the 
model  over  different  levels  as  listed: 

coefficients  -  seven  levels 

goodness  -  five  levels 

adjustment  -  three  levels 

scoring  method  -  two  levels 

The  concern  was  to  determine  if  any  of  the  treat¬ 
ments  significantly  changed  the  mean  frequency  of  selection 
of  the  "best"  proposal. 

A  suitable  statistical  technique  for  determining 
the  significance  of  any  observed  change  over  a  number  of 
observations  of  different  levels  of  treatments  is  Analysis 
of  Variance  (ANOVA) ,  (11:526). 


The  assumptions  of  ANOVA  are  that: 

(1)  The  probability  distributions  of  the  depend¬ 
ent  variables  are  normal. 

(2)  The  variance  of  the  dependent  variable  is 
constant . 

(3)  The  samples  are  independent. 

Regarding  the  assumption  of  normality,  the  variable 
of  interest  in  the  experiment  was  the  number  of  times  the 
"best"  proposal  was  selected,  i.e.,  the  result  of  n 
Bernoulli  trials  of  which  the  outcome  was  either  "selected" 
or  "not  selected".  The  distribution  of  such  a  series  of 
events  has  the  binomial  probability: 

'<*>  -  C) 

where  n  =  sample  size 

x  =  the  number  of  events  of  interest  in  n 

and  p  =  probability  of  occurrence  of  an  event 
of  interest 

(11:137) 

However,  when  the  sample  size  n  is  reasonably 
large  (n>30),  the  binomial  probability  distribution  can 
be  approximated  by  a  normal  probability  distribution 
(l 1:216),  and  it  has  been  shown  (l4:6l)  that  a  moderate 
departure  from  normality  has  little  effect  on  the  test  of 
significance  of  ANOVA. 
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A  preliminary  scanning  of  the  computer  model 
output  suggested  that  constancy  of  variance  was  a  reason¬ 
able  assumption.  It  was  decided  to  proceed  to  ANOVA  on 
that  basis  and  use  the  Cochran's  "C"  procedure  provided 
with  the  SPSS  program  to  test  the  assumption  after  the 
event  ( 12:430). 

The  sample  data  of  the  computer  model  was  stat¬ 
istically  independent  to  the  extent  of  the  independence 
of  the  pseudo-random  number  generator.  The  condition  of 
independence  was  regarded  to  be  satisfied  for  the  purposes 
of  ANOVA  for  all  treatments  except  for  the  treatment 
"method",  (color  or  numerical  scoring).  For  simulation 
of  "method" ,  the  treatments  were  successively  applied  to 
the  same  sets  of  basic  data.  ANOVA  was,  therefore,  chosen 
as  the  means  by  which  to  examine  the  treatment  effects 
of  "weighting  coefficients",  "goodness",  and  "adjustment 
parameter" . 

As  the  treatment  "method"  involved  only  two  levels 
of  treatment  (color  or  numbers),  it  was  appropriate  to 
apply  the  t-test  for  population  mean  differences  between 
matched  samples  to  study  the  effect  of  "method"  (11:320). 

Sample  Size 

It  was  desired  to  have  a  sample  size  so  that  the 
ANOVA  would  provide  information  about  the  discrimination 
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of  the  computer  model  with  10  per  cent  confidence  level 
with  10  per  cent  accuracy  of  estimation  of  the  frequency 
of  selection  of  the  "best"  proposal. 

For  a  binomial  probability  distribution,  the 
sample  size  can  be  estimated  by: 

n  = 

(15s 191 ) 

where  Z(ot/2)  is  the  two-tailed  normal  statistic 
for  the  desired  confidence  level,  and 

d  is  the  difference  between  the  true 
probability  of  selection  and  the  estimate. 

For  the  experimental  requirements,  n  was  calculated 
to  be  68 j  5  computer  runs  of  80  simulations  were  selected 
to  provide  an  adequate  data  base  for  evaluation  of  results. 

ANOVA  Test  Procedure 

The  computer  model  outputs  were  first  tested  to 
determine  the  significance  of  the  different  treatments 
when  applied  separately.  SPSS  procedure  ONEWAY  was 
employed.  There  are  two  steps  involved  in  using  this 
technique: 

(l)  Test  the  hypotheses 

H0  :  There  is  no  difference  in  the  mean 
proportion  of  proposals  number  1 
selected  between  different  levels  of 
the  treatment  being  studied. 


(CT/2) 

4d2 
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There  is  a  difference  in  the  mean  prop¬ 
ortion  of  proposals  number  1  selected 
between  different  levels  of  the  treat- 


H 


1 


raent  being  studied. 

The  decision  rule  for  the  test  is: 

if  F* <  F(0.9»r-1 ,nt~r) ,  conclude  He,  other 


wise  conclude  H1  ( 1 1 s 53 5 ) • 

„„  treatment  mean  sauare 

Where  F*  =  ...  —  - - - 

error  mean  square 

and  r  =  the  number  of  treatment  levels 
n  =  total  number  of  observations . 

The  value  of  F*  is  provided  as  part  of  the 
SPSS  output. 

(2)  If  the  test  shows  a  difference  between  means, 
analyze  the  ranges  within  which  the  differ¬ 
ences  lie.  Duncan's  multiple  range  test 
provided  in  the  SPSS  package  (12:427)  is 
suitable  for  this  purpose. 

A  multiple  ANOVA  analysis  was  then  conducted  of  the 
significant  treatments  to  examine  interaction  effects  over 
the  range  of  treatments  when  color  scoring  was  used  in  the 
model . 


ONEWAY  ANOVA  Results 

Eight  data  sets  were  selected  for  ONEWAY  analysis 
to  obtain  a  feel  for  the  separate  treatment  effects  on  mean 
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proportion  of  proposals  number  one  selected.  The  results 
are  presented  in  Table  IXX. 

The  results  in  Table  III  show  that,  for  the  para¬ 
meter  sets  tested,  the  treatment  "weighting  coefficients" 
was  not  significant  at  the  0.1  level  in  determining  the 
frequency  with  which  the  "best"  proposal  was  selected. 

Both  "goodness  set"  and  "adjustment  parameter"  (B0)  were 
significant  treatments  which  affected  the  outcome  of  the 
selection. 

The  results  of  the  Duncan's  multiple  range  tests 
are  shown  in  Table  IV. 

The  results  show  that  when  the  numerical  scoring 
process  was  applied,  the  mean  frequency  of  selection  of 
the  "best"  proposal  was  significantly  different  for  each 
goodness  set  of  proposals.  The  frequency  of  selection  of 
the  "best"  proposal  increased  as  the  quality  difference 
between  the  proposals  increased. 

A  similar  result  was  shown  when  color  scoring  was 
used,  except  that  the  frequency  of  selection  of  the  "best" 
proposal  in  each  goodness  set  was  consistently  less  than 
when  number  scoring  was  used  and  that  the  frequency  of 
selection  of  the  "best"  proposal  was  not  significantly 
different  when  the  difference  between  the  "best"  and  "next 
best"  proposals  was  large. 
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TABLE  IV 

TABLE  SHOVING  HOMOGENEOUS  SUBSETS  OF  TREATMENTS 

Test 
No. 

1  T'ment  Level 
Mean 
Subset 

2  T'ment  Level 
Mean 
Subsets 


3  T'ment  Level  1  2  3  4  5 

Mean  35.0  43.8  53.8  64.8  69.8 

Subsets  _ 


1 

2 

3 

4 

5 

6 

7 

52.4 

53.6 

52.2 

53.8 

54.4 

54.6 

55.0 

1 

2 

3 

4 

5 

6 

7 

45.0 

47.6 

47.4 

49.6 

49.8 

52.8 

54.2 

4  T'ment  Level  12345 
Mean  31.6  42.8  49.6  64.0  66.2 

Subsets  _ 


5  T'ment  Level  123 
Mean  52.0  42.8  4l.O 

Subsets  _ 


6  T'ment  Level  123 
Mean  58.6  49.6  48.0 

Subsets  _ 


7  T'ment  Level  123 
Mean  69.4  64.0  6 2.0 

Subsets  _ 


8  T'ment  Level  123 
Mean  71.4  66.2  65. 0 

Subsets  _ 


With  regard  to  the  ONEWAY  analysis  of  the  treatment 
"adjustment  parameter"  (B0),  included  when  color  scoring 
was  used,  treatment  level  1  (B0  =  +7)  was  found  to  cause 
a  significantly  different  result  in  the  selection  of  the 
"best"  proposal  at  all  goodness  levels.  There  was  no 
significant  difference  between  the  effects  of  treatment 
levels  2  and  3  (B0  =  0  and  B0  =  -7  respectively). 

The  values  obtained  for  P  in  the  Cochran's  C  test 
show  that  in  all  cases  the  assumption  of  homogeneity  of 
variances  was  met  at  the  0.1  level,  justifying  the  valid¬ 
ity  of  the  ANOVA  approach. 

Difference  Between  Color-Scored  and  Numerically-Scored 

Results  (t-Test) 

The  concept  of  value  building  by  using  numerical 
component  scores  and  weights  does  not  include  the  adjust¬ 
ment  factor,  B0.  It  was  therefore  appropriate  for  the 
purpose  of  this  test  to  compare  numerical  scores  with  color 
scores  only  at  the  B0  =  0  level. 

The  purpose  of  the  t-test  was  to  determine  if  the 
frequency  of  selection  of  the  "best"  proposal  was  signif¬ 
icantly  greater  at  the  0.1  level  when  number  scores  were 
used  than  when  color  scores  were  used. 

If  the  mean  score  by  numbers  is  M  and  the  mean 

n 

score  by  colors  is  Mc>  then  the  test  hypothesis  is: 
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H,  :  M  >  M 
Inc 

and  the  decision  procedure  using  SPSS  output  (l2:27l)  is: 

if  the  one-tailed  probability  is  larger  than  cx 
do  not  reject  H0 . 

The  t-test  was  conducted  over  the  range  of  goodness 
sets  2  to  5  and  at  level  4  and  ot  =  . 1 .  The  results 
are  presented  at  Table  V  in  which  H0  is  concluded  for 
goodness  sets  2  and  4  and  H..  for  goodness  sets  3  and  5* 

Multiple  ANOVA  (MANOVA) 

The  ONEWAY  ANOVA  test  results  showed  the  effects  of 
treatments  "goodness"  and  "adjustment  parameter"  (B0)  to 
be  significant  at  the  0.1  level  for  the  fixed  parameter 
values  tested.  Treatment  "weighting  coefficients"  (B^ 
set)  was  found  to  be  ineffective  at  0.1  level  of  confidence 
for  numerical  scoring  and  B0  =  0  and  goodness  set  number  3. 
However,  when  the  parameter  "colors"  was  included  in  the 
test  for  significance  of  treatment  "weighting  coefficients", 
the  value  of  F*  (1.845)  was  close  to  the  value  of  F(2.00). 

It  was  considered  advisable  to  include  "weighting  co¬ 
efficients"  as  a  treatment  in  the  MANOVA  in  case  it  became 
significant  at  the  extremes  of  range  of  treatments  or  when 
applied  in  conjunction  with  other  treatments. 


50 


TABLE  V 
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The  SPSS  ANOVA  sub-program  is  designed  to  handle 


MANOVA  for  factorial  experimental  designs. 

Since  the  ONEWAY  test  for  treatment  "goodness  set" 
yielded  very  large  values  of  F*,  it  was  likely  that  the 
effect  of  varying  "goodness  set"  would  overwhelm  the 
effects  of  treatments  "adjustment  parameter"  and  "coeffic¬ 
ient  set"  for  goodness  sets  2  through  4.  A  symmetrical 
factorial  design  was  chosen  with  three  levels  each  of: 
"adjustment  parameter";  B0  =  +7»0,-7 
and  "coefficient  set";  set  No.  1,  set  No.  4  and 
set  No.  7. 

The  multiple  classification  analysis  (MCA)  option 
of  the  SPSS  ANOVA  program  was  used  to  provide  an  indica¬ 
tion  of  the  magnitude  of  the  effect  of  each  treatment.  The 
outputs  are  presented  at  Appendices  C1-C8. 

Summary  of  Results  of  Analyses  of  Model  Output 

The  ONEWAY  test  results  show  that  the  treatments 
"goodness  set"  and  "adjustment  parameter"  are  significant 
at  the  0.1  level  in  determining  the  probability  of  selec¬ 
tion  of  proposal  number  1 .  "Coefficient  set"  was  not  a 
significant  treatment  for  either  number  or  color  scoring 
at  goodness  set  number  3  and  adjustment  parameter  2 
(B0  =  0). 

Further  analysis  of  the  treatments  "coefficient 
set"  and  "adjustment  parameter"  taken  conjointly  in  a 
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two-way  MANOVA  for  goodness  sets  2  through  4  when  numerical 
scoring  was  used,  show  different  joint  effects  of  treat¬ 
ments  "coefficient  set" and  "adjustment  factor"  as  the 
level  of  goodness  set  is  increased,  i.e.,  as  the  difference 
in  quality  between  the  "best"  proposal  and  the  "second 
best"  proposal  increases. 

When  the  difference  between  proposals  is  small, 
"adjustment  factor"  (B0)  is  the  significant  external 
treatment.  Large  positive  values  of  B0  increase  the  fre¬ 
quency  of  selection  of  the  "best"  proposal.  At  goodness 
set  2  (10%  difference  between  proposals),  B0  explained 
0.25  of  the  selection  preference,  whereas  the  coefficient 
set  explained  only  .04  of  the  selection  preference. 

However,  both  treatments  accounted  for  a  relatively  small 
part  of  the  selection  and  a  large  variance  of  outcomes 
was  predicted. 

At  goodness  set  3  (20%  difference  between  propos¬ 
als),  coefficient  set  and  adjustment  parameter  each 
explained  about  .16  of  the  selection  preference  with 
still  a  relatively  large  variance  of  outcomes. 

At  goodness  set  4  (30%  difference  between  propos¬ 
als),  coefficient  set  became  the  dominant  reason  for 
selection  preference,  explaining  0.50  of  the  outcome 
while  adjustment  parameter  explained  0.12  of  the  outcome. 

At  goodness  set  5  (40%  difference  between  propos¬ 
als),  coefficient  set  was  even  more  dominant,  explaining 
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0.56  while  adjustment  factor  still  explained  0.12  of 
the  outcome.  The  variance  of  selection  due  to  unex¬ 
plained  factors  of  the  model  was  reduced  as  the  gap 
between  "best"  and  "next  best"  proposal  increased. 

The  direction  of  effects  of  the  treatments  was 
also  worthy  of  note.  Large  positive  values  of  adjustment 
factor  (B0)  increased  the  probability  of  selection  of 
the  "best"  proposal.  Negative  values  of  B0  reduced 
the  probability  of  selection.  Coefficient  sets  with 
small  differences  in  component  weights  forced  selection 
toward  the  "best"  proposal  while  larger  differences  in 
component  weights  resulted  in  greater  variance  of 
selection  of  proposals. 

The  results  of  t-tests  for  the  effect  of  treat¬ 
ment  "method"  (color  or  numerical  scoring)  were  less 
conclusive.  At  goodness  sets  3  and  5,  the  discriminat¬ 
ing  power  of  numerical  scoring,  as  modeled,  was  sig¬ 
nificantly  greater  than  the  power  of  color  scoring  at 
the  0.1  level.  There  was  no  significant  difference 
between  the  two  scoring  methods  at  goodness  sets  2  and 
4. 
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CHAPTER  IV 


INTERVIEWS  WITH  SOURCE  SELECTION  PRACTITIONERS 

To  further  examine  the  underlying  nature  of  the 
source  selection  decision-making  process,  structured 
interviews  were  conducted  with  source  selection  practi¬ 
tioners  and  administrators  in  an  attempt  to  identify 
their  perceptions  of  the  models  in  the  field.  The 
Aeronautical  Systems  Division’s  Directorate  of  Contract¬ 
ing  and  Manufacturing,  originally  established  as  point 
of  contact  in  this  research  effort,  provided  a  listing  of 
selected  ASD  personnel  who  were  at  the  time,  or  had 
recently  been,  engaged  in  different  aspects  of  source 
selection.  Thirty-one  personnel  were  interviewed.  All 
had  been  involved  in  at  least  one  of  the  many  different 
functions  of  source  selection,  including  acquisition 
policy  and  procedure  management,  SSA,  SSA  advisor,  SSAC 
member,  SSEB  Chairman,  item  captain,  Acquisition  Logistic 
Division  (ALD)  representative,  program  manager,  principal 
contracting  officer  (PCO),  and  general  contracting, 
pricing,  buying  and  manufacturing  participants. 

An  interview  guide  was  prepared  to  ensure  consist 
ency  of  approach  in  the  research.  A  copy  of  the  guide  is 
attached  at  Appendix  D.  The  interviews  were  designed  to 
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try  to  obtain  an  overall  view  of  the  source  selection 
decision-making  process  from  a  participant's  viewpoint 
and  to  assist  in  identifying  the  reality  of  the  process 
as  it  is  applied  in  practice  against  the  theoretical 
models  of  multi-attribute  decision-making.  Discussions 
were  centered  around  the  following  areas:  effectiveness 
and  efficiency  of  existing  source  selection  decision 
procedures,  relative  merits  of  numerical  and  color-coding 
schemes  of  scoring  the  results  of  evaluations,  influence 
of  Contractor  Inquiries  (CIs)  and  Deficiency  Reports  (DRs) 
on  the  decision  process,  and  ways  of  improving  the  source 
selection  decision  process. 

Effectiveness  and  Efficiency  of  the  Process 

There  was  little  agreement  on  what  the  ultimate 
objective  of  source  selection  should  be,  although  the 
majority  of  the  personnel  interviewed  agreed  that  existing 
source  selection  decision  procedures  assured  the  effective¬ 
ness  of  the  process  in  attaining  its  perceived  objective. 
Responses  included  the  following: 

"a  mechanism  to  appear  as  objective  as  possible 
in  selecting  a  source  while  protecting  against  protests 
and  complaints" 

"to  get  best  contractor  at  best  price" 

"to  de-select  other  offers  to  be  able  to  withstand 
protests" 
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"to  get  technically  best  contractor1* 

"to  select  best  source  for  the  government,  all 
factors  considered" 

"to  give  you  a  good  insight  before  you  commit 
yourself" 

"to  select  best  supplier  at  best  price,  if  price 
is  one  consideration" 

"to  be  fair  in  selecting  a  source  able  to  perform" 

"to  get  the  best  capability  in  meeting  the  needs 
of  the  Air  Force  in  accordance  with  the  requirements 
of  the  solicitation". 

Statements  such  as  these  clearly  show  a  lack  of 
agreement  and  possibly  misunderstanding  among  personnel 
interviewed  regarding  the  purpose/ob jective  of  the  source 
selection  process.  A  clear  understanding  of  the  ultimate 
objective  of  the  process  by  its  participants  is  essential 
to  ensure  effective  evaluation  of  proposals  and  results 
which  meet  the  ultimate  objective  of  the  source  selection 
process . 

Far  greater  agreement  was  found  among  those  inter¬ 
viewed  when  asked  about  the  effectiveness  of  the  process  in 
achieving  the  perceived  overall  objective.  A  large  majority 
agreed  that  the  process  is  usually  effective  in  meeting 
its  stated  objectives,  and  that  the  right  contractor  is 
selected  in  almost  all  cases.  Some  concern  was  expressed 
though,  regarding  normative  political  override.  Source 
selection  decisions  are  sometimes  made  on  political  con- 
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siderations  without  adequately  quantifying  the  risk  of 
program  failure. 

A  number  of  factors  seems  to  hamper  the  effective¬ 
ness  of  the  existing  source  selection  process.  Among 
factors  cited  was  the  effect  that  funding  constraints 
have  on  the  source  selection  decision.  During  the  last 
decade,  budgeting  has  been  a  major  external  influence  on 
the  process,  creating  "a  temptation  to  make  the  low  offer 
appear  to  meet  the  requirements"  through  extensive  use 
of  Cls  and  DRs. 

The  massive  amount  of  data  with  which  evaluators 
are  confronted  when  evaluating  proposals  was  seen  to  be  a 
major  factor  in  preventing  a  truly  effective  process.  It 
was  said  that  evaluators  usually  find  it  difficult  to 
filter  out  the  data  in  order  to  identify  and  be  able  to 
assess  the  key  issues.  It  appears  that  source  selection 
evaluations  are  being  made  with  an  excessive  amount  of 
data--far  more  than  that  which  is  needed--obscuring  the 
important  issues  and  preventing  decision-makers  from 
effectively  evaluating  them. 

Most  of  the  personnel  interviewed  expressed  concern 
about  the  inefficiency  of  the  source  selection  process. 

Many  said  that  they  considered  the  process  to  be  grossly 
inefficient,  due  mainly  to  the  large  number  of  people 
involved  in  the  evaluation  stages,  the  excessive  amount  of 


time  taken  up  by  evaluations,  and  the  large  amounts  of 
data  encountered  in  proposals.  The  large  and  detailed 
RFPs  sent  out  to  industry  seem  partly  to  be  the  cause  of 
much  of  this  ineff iciency .  The  RFPs  force  offerors  to 
generate  large  amounts  of  data  in  support  of  their  pro¬ 
posals  and  make  evaluation  a  time-consuming,  extremely 
complex  process  which  requires  many  evaluators  in  order 
to  sort  out  the  data. 

More  than  half  of  the  respondents  said  that  the 

source  selection  process  involves  far  too  many  people. 

Lack  of  expertise  and  evaluating  experience  was  cited  by 

some  as  contributing  to  the  inefficiency  of  the  process. 

\ 

It  also  appears  that  the  government  spends  a  dispropor¬ 
tionately  large  amount  of  resources  in  obtaining  a  small 
system  in  relation  to  that  which  it  spends  in  acquiring 
a  major  system.  The  need  to  streamline  the  process  was 
emphasized.  Some  suggested  that  a  small  group  of  10  to 
15  qualified  evaluators  could  reach  a  decision  as  accept¬ 
able  as  that  made  by  a  large  number  of  evaluators. 

Some  concern  was  expressed  regarding  the  amount 
of  resources  spent  in  areas  which  did  not  influence  the 
final  decision.  Much  emphasis  is  placed  on  certain  areas 
of  proposals,  e.g.,  management.  The  effort  evaluators 
put  into  these  areas  seems  unwarranted  when  the  output  of 
such  evaluations  fails  to  have  an  impact  on  the  decision 
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process.  It  was  observed  that  there  is  a  trend  toward 
increasing  the  number  of  management  evaluation  items. 

While  some  of  the  perceived  inefficiency  attrib¬ 
uted  to  the  process  may  be  caused  by  the  need  to  document 
every thing  in  order  to  have  a  sound  defense  against 
potential  protests,  such  a  fear  of  protests  appears  to 
be  unfounded.  Less  than  4  per  cent  of  contracts  awarded  by 
ASD  result  in  protests,  with  the  majority  of  the  protests 
being  shown  to  be  without  foundation. 

In  summary,  it  appears  that  a  number  of  factors 
cause  many  people  to  be  involved  in  source  selection. 
However,  the  process  seems  to  have  worked  effectively, 
and  the  desired  results  have  been  achieved  as  well  in 
those  cases  where  strong  management  has  insisted  on  a 
reduced  number  of  evaluators. 

The  Scoring  Process 

AFR  70-15  provides  broad  guidance  on  source  sel¬ 
ection  decision  procedures.  It  discusses  the  use  of  both 
numerical  and  color-coding  schemes  of  scoring  the  results 
of  evaluations,  supported  by  narrative  statements.  ASD 
regulations  encourage  the  use  of  color-coded  and  narrative 
assessments,  and  numerical  scoring  has  not  been  formally 
used  in  ASD  since  June  1972.  In  an  attempt  to  identify 
the  strengths  and  weaknesses  of  both  the  numerical  and 
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color-coding  techniques,  personnel  Interviewed  were  asked 
to  comment  on  the  relative  merits  of  each  approach. 

While  about  one-half  of  those  interviewed  expressed 
their  preference  for  the  use  of  colors,  one-third  Indicated 
that  both  methods  were  equally  effective  in  assessing 
proposals,  with  a  few  personnel  showing  a  preference  for 
the  numerical  scoring  technique.  The  preference  for  the 
color-coding  approach  seemed  to  be  based  on  the  concept 
of  providing  an  integrated  assessment  which  would  highlight 
the  strengths,  weaknesses,  and  risks  of  each  proposal  and 
allow  the  SSA  greater  latitude  to  exercise  judgment. 

Under  the  numerical  scoring  system,  the  SSA  felt 
constrained  to  accept  the  numerical  results,  and  a  deci¬ 
sion  to  select  a  source  other  than  the  one  with  the 
highest  scoring  proposal  was  difficult  to  justify.  Com¬ 
ments  were  also  expressed  that  source  selection  is  partly 
a  qualitative  judgment  process  which  is  sometimes  hard  to 
quantify  and  creates  difficulty  in  arriving  at  an  agreed 
number,  whereas  agreement  is  much  more  easily  reached 
using  color  scores.  Areas  such  as  past  performance  and 
management  are  sometimes  difficult  to  weigh  and  score  with 
numbers  giving  an  unwarranted  degree  of  precision,  while 
color-coding  provides  a  clearer  overall  picture  to  the 
decision-maker. 
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Individuals  who  expressed  the  view  that  both 
approaches  were  equally  effective  and  would  serve  to 
accomplish  essentially  the  same  purpose,  indicated  that 
the  important  thing  is  to  conduct  a  balanced  evaluation 
which  ensures  key  areas  are  identified  appropriately  and 
evaluated  properly. 

It  was  frequently  stated  that  in  the  more  object¬ 
ive  areas,  e.g.,  technical,  evaluators  made  initial  scores 
on  a  numerical  scale.  They  then  converted  these,  using 
cut-off  valuer  to  color  scores  to  fit  in  with  the  source 
selection  plan. 

Those  who  preferred  the  numerical  approach  said 
that  numbers  provided  a  quicker  reaction  to,  and  identif¬ 
ication  of,  slight  differences  between  similar  proposals. 
The  numerical  scoring  technique  appears  to  yield  a  more 
discrete  and  finer  identification  of  differences  at  the 
attribute  level;  something  color-coding  fails  to  do.  It 
forces  the  attribute  evaluator  to  commit  himself  to  a 
firm  decision.  In  addition,  numerical  scoring  allows  the 
weighting  of  issues  to  be  precisely  identified  in  advance 
of  scoring  according  to  their  relative  importance  as 
established  in  the  source  selection  plan.  Conversely, 
they  said  that  color-coding  introduces  a  degree  of  un¬ 
certainty  and  encourages  political  maneuvering. 
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In  discussing  numerical  scoring,  a  variety  of 
perceptions  of  the  "cut-off"  level  of  discrimination  of 
numerical  scores,  one  to  the  other,  and  when  compared  to 
a  standard,  was  discovered.  Some  respondents  said  that 
an  absolute  difference  between  scores  was  a  sufficient 
basis  on  which  to  make  a  decision.  Most  who  gave  an 
opinion  said  that  a  difference  of  1  to  2  per  cent  between 
scores  was  significant.  Less  than  that,  other  (subjective) 
considerations  would  come  into  the  decision.  About  half 
the  respondents  felt  they  could  not  give  an  opinion,  and 
one  experienced  officer  said  that  if  numerical  scores 
were  used  in  systems  source  selections,  he  would  not 
consider  score  differences  of  less  than  10  per  cent  to  be 
significant.  Respondents  frequently  said  that  in  many 
cases  ASD  was  concerned  with  buying  concepts  which  did  not 
lend  themselves  to  highly  objective  scoring. 

When  evaluating  proposals  at  the  SSEB  level, 
proposals  should  be  compared  against  standards  established 
in  the  solicitation  document.  A  tendency  to  compare 
proposals  with  each  other  at  this  level,  rather  than 
against  standards,  as  required  by  regulations  (l7*P«3-*0 
was  expressed  by  some  of  those  interviewed. 

Contractor  Inquiries  and  Deficiency  Reports 

A  considerable  amount  of  effort  is  spent  by  source 
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selection  personnel  in  the  preparation  of  CIs  and  DRs  as 
the  means  of  communicating  with  offerors,  to  provide  for 
clarification  of  certain  aspects  of  proposals,  and  to 
identify  specific  parts  of  proposals  which  fail  to  meet 
the  government’s  minimum  requirements.  This  procedure 
allows  the  offerors  to  correct  deficiencies  found  by 
evaluators.  Almost  every  one  of  the  personnel  interviewed 
agreed  that  although  the  Cl/DR  process  is  time-consuming 
and  usually  prolongs  the  evaluations,  it  is  essential  to 
obtaining  a  satisfactory  contractual  arrangement  and  is 
significant  in  influencing  the  decision  process. 

Responses  indicated  a  frequent  excess  of  CXs. 

This  was  partly  due  to  the  failure  of  RFPs  to  be  definitive 
in  some  areas.  The  excess  was  also  attributed  to  the 
reluctance  of  evaluators  to  make  a  subjective  judgment, 
and  attempts  to  obtain  a  defensible,  documented  position. 

It  was  suggested  by  some  respondents  that  more  direct 
talks  with  offerors  would  help  to  reduce  the  number  of  CIs 
originated  and  eliminate  much  of  the  paperwork  created 
during  the  process.  It  was  observed  that,  in  those  cases 
where  ASD  had  used  the  four-step  solicitation  process 
(20:4)  there  was  a  large  reduction  in  the  use  of  CIs. 

DRs  were  considered  to  be  far  more  critical  in  influencing 
the  decision  process,  since  these  documents  allow  evalua¬ 
tors  to  determine  how  well  final  offers  meet  the  govern- 
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merit's  requirements.  The  most  important  ones  are  usually- 
highlighted  under  "strengths  and  weaknesses"  in  evaluation 
reports  to  the  higher  levels  of  decision-making. 

Although  AFR  7 0-15  requires  that  proposals  only 
be  scored  as  originally  submitted  to  encourage  the  best 
initial  proposals,  it  was  found  that,  in  practice,  prop¬ 
osals  were  often  rescored.  A  review  of  three  source 
selection  case  histories  in  ASD,  together  with  responses 
obtained  during  the  interviews ,  indicated  that  proposals 
are  frequently  rescored  after  the  Cl/DR  process  is  com¬ 
pleted.  It  appears  that  further  clarification  and 
guidance  regarding  rescoring  of  proposals  may  be  required 
to  ensure  that  a  fair  and  consistent  approach  is  used. 

Improvements  Suggested  by  Interviewees 

It  was  agreed  that  the  existing  source  selection 
process  is  usually  effective  in  selecting  the  proposal(s) 
which  best  meets  the  government's  cost,  schedule  and  per¬ 
formance  requirements,  considering  that  what  is  being 
evaluated  usually  is  an  offeror's  future  performance  of 
something  which  is  essentially  innovative.  However,  a 
great  majority  of  the  personnel  interviewed  saw  much 
room  for  improvement  of  the  process.  The  discussion  that 
follows  concentrates  on  those  areas  suggested  to  have  the 
greatest  potential  for  improvement  of  the  overall  process. 
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A  need  to  integrate  the  source  selection  activity 
with  that  of  preparing  RFPs  was  expressed.  A  great  part 
of  the  source  selection  plan  and  process  is  determined  by 
the  way  the  RFP  was  written.  It  was  said  that  closer 
coordination  between  source  selection  personnel  and  those 
responsible  for  the  preparation  of  RFPs  would  help  ensure 
that  more  definitive  and  concise  requirements  go  out  to 
industry.  This  contact  would  result  in  more  compact  and 
precise  proposals  which  would  serve  to  reduce  the  tre¬ 
mendous  amounts  of  data  with  which  evaluators  are  presently 
being  confronted,  would  allow  the  significant  aspects  and 
key  issues  to  surface  sooner,  and  would  provide  for  a 
more  effective  and  efficient  evaluation.  Some  respondents 
suggested  that  the  size  of  proposals  should  be  controlled 
by  defining  in  the  RFP  the  number  of  pages  of  submission 
allowed. 

Further  streamlining  of  the  process  was  suggested 
to  help  make  it  more  efficient.  It  was  suggested  that  a 
group  of  well-qualified  and  experienced  personnel  with 
broad  knowledge,  complemented  with  competent  technical 
advisors,  would  result  in  a  reduced  number  of  evaluators 
and  a  shorter  time  required  to  assess  proposals.  It  was 
said  that  the  source  selection  experience  of  evaluators 
must  be  improved  and  more  specific  guidance  provided  for 
first-time  evaluators.  The  lack  of  a  viable  training 
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program  in  source  selection  procedures  for  evaluators 
with  no  previous  experience  in  source  selection,  makes 
it  a  difficult  task  for  those  personnel  who  have  to  learn 
the  procedures  while  on  the  job.  This  shortcoming  results 
in  much  unproductive  time  and  decreased  efficiency. 

An  awareness  that  many  of  the  problems  which 
surface  during  the  performance  of  a  contract  are  related 
to  the  contractor’s  data  and  cost  tracking  systems  has 
directed  an  increased  emphasis  on  the  management  area  of 
proposals  during  evaluations.  Respondents  indicated  that 
although  a  considerable  amount  of  effort  is  spent  in  this 
area,  it  seldom  influences  the  decision  process.  An 
improved  approach  for  assessing  the  management  area  of 
proposals  in  a  more  realistic  way  was  felt  to  be  necessary, 
with  increased  emphasis  being  placed  on  a  prospective 
contractor's  past  performance. 

It  was  felt  that  there  was  a  need  to  develop  a 
better  way  of  linking  the  cost  and  technical  evaluations 
together  in  order  to  obtain  a  realistic  cost-benefit 
analysis.  Other  suggestions  in  this  area  dealt  with  the 
need  to  bring  together  the  assessments  of  the  Cost  Panel 
and  the  Contract  Def initization  Group  at  some  point  during 
the  process  to  provide  a  better  overall  picture  when 
considering  tradeoffs  between  cost  and  technical  require¬ 
ments  . 
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Increased  use  of  the  abbreviated  procedures  for 
source  selection  was  advocated.  In  the  abbreviated  pro¬ 


cedure  a  Source  Selection  Evaluation  Committee  (SSEC) 
assumes  the  responsibilities  of  the  SSAC  and  SSEB.  This 
resulted  in  a  more  efficient  process.  It  seemed  evident 
to  some  interviewees  that  frequently  the  SSAC  failed  to 
apply  the  judgment  required  in  a  comparative  analysis  of 
proposals,  and  merely  served  as  a  means  of  filtering  the 
SSEB  evaluation  results  to  the  SSA.  It  was  also  felt 
that  some  of  the  more  formal  requirements  for  source 
selections  on  lower-dollar  acquisitions  could  be  eliminated, 
improving  the  efficiency  of  the  process  without  impacting 
on  its  effectiveness. 

A  need  to  rescore  proposals  after  the  DR  process 
is  completed  was  thought  by  many  to  be  an  essential  pro¬ 
cedure  to  ensure  an  optimum  decision.  Scoring  proposals 
as  originally  submitted  and  as  corrected  seemed  to  be  the 
only  way  to  conduct  a  realistic  appraisal. 

During  the  course  of  the  research,  it  became  evid¬ 
ent  that  some  source  selections  departed  significantly 
from  the  guidance  provided  in  regulations;  a  fact  which 
was  felt  by  some  people  interviewed  to  cause  some  of  the 
inefficiency  attributed  to  the  process.  This  was  thought 
to  be  partly  due  to  the  lack  of  recent  and  current  guide- 
ance  in  the  field.  AFR  70-15,  the  primary  document  for 
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establishing  policy  and  procedures  for  the  conduct  of 
source  selections  in  the  Air  Force,  is  now  five  years 
old,  outdated  and  has  been  under  revision  for  over  a  year. 

It  was  hoped  that  when  the  new  issue  of  AFR  70-15  is 
published,  it  will  provide  more  specific  guidance  for 
the  conduct  of  source  selections. 

Some  concern  was  expressed  that  major  contractors 
have  developed  an  ability  to  submit  high  scoring  bids  which 
makes  it  difficult  to  assess  proposals  which,  on  paper, 
appear  to  be  fairly  similar.  Evaluation  then  becomes  a 
task  of  determining  whether  the  offeror  is  able  to  do  what 
he  says  he  can  do,  rather  than  making  an  objective  techni¬ 
cal  decision.  As  this  seems  to  be  the  case  during  many 
formal  source  selections,  it  becomes  critical  to  provide 
the  SSA  with  objective  information  on  which  to  make  a 
rational  decision  which  will  reduce  the  risk  of  cost 
overruns  and  program  slippages. 

Summary 

The  interviews  provided  a  good  insight  of  the  source 
selection  process  as  it  is  presently  applied,  and  identified 
a  number  of  difficulties  perceived  by  source  selection 
participants. 

Even  though  respondents  agreed  that  the  process 
was  effective  in  achieving  its  perceived  objective,  there 
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was  little  agreement  as  to  what  that  objective  should  be. 
Concern  was  expressed  regarding  the  inefficiency  of  the 
process.  This  inefficiency  was  attributed  to  the  large 
number  of  people  involved  in  source  selections,  the 
excessive  amount  of  time  taken  up  by  evaluations,  and 
the  massive  amount  of  data  with  which  evaluators  are 
confronted. 

Views  regarding  the  techniques  used  for  scoring 
proposals  provided  a  wide  range  of  opinions  of  the 
relative  merits  of  each  approach.  Preference  for  numer¬ 
ical  or  color  scoring  methods  was  divided. 

Although  it  was  evident  that  the  Cl  and  DR 
processes  are  time-consuming,  they  were  considered  to 
be  essential  and  very  significant  in  influencing  the 
decision  process  and  in  making  a  satisfactory  contract. 

Interviewees  agreed  that  there  was  room  for 
improvement  of  the  process.  Their  responses  suggest  some 
approaches  for  accomplishing  that  objective. 

Source  selections  in  ASD  cover  a  wide  range  of 
acquisitions  of  varying  degrees  of  complexity  and  maturity 
of  concept.  However,  there  is  some  evidence  that  the 
process  is  not  always  applied  with  sufficient  judgment 
and  that  departures  from  policy  and  procedures  occur. 


CHAPTER  V 


CONCLUSIONS  AND  RECOMMENDATIONS 

This  study  was  directed  toward  identifying  the 
process  of  source  selection  as  practised  in  ASD.  The 
methodology  of  source  selection  was  simulated  through  a 
computer  model.  A  perspective  of  the  process  was  devel¬ 
oped  through  a  review  of  the  procedural  guidance  and  a 
series  of  interviews  with  ASD  source  selection  personnel. 
This  chapter  summarizes  the  findings  of  the  study  and 
compares  them  with  some  theoretical  concepts  to  develop 
a  descriptive  evaluation  of  the  ASD  source  selection 
process.  Finally,  recommendations  are  made  which  may 
contribute  to  the  improvement  of  the  management  of  source 
selection. 


Source  Selection  Methodology 

The  analysis  of  source  selection  cases  in  which 
color  or  narrative  scoring  methods  were  used  demonstrated 
the  possibility  of  evaluators  incorporating  an  adjustment 
parameter  (B0)  into  the  value  building  process  when 
aggregating  a  group  of  lower-level  attribute  scores. 

The  effect  of  introducing  negative  values  of  B0 
into  the  simulation  model  was  to  reduce  the  discrimination 
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of  the  process  in  selecting  the  "best"  proposal  in  terms  of 
the  evaluation  criteria.  Positive  values  of  B0  biased  the 
scores  in  favor  of  the  "best"  proposal.  The  B0  effect  was 
greatest  when  the  difference  between  proposals  was  small. 
More  cases  of  negative  values  of  B0  were  observed  than 
positive  values  suggesting  a  "wash-out"  of  the  component 
evaluations  in  those  cases. 

As  might  be  expected  from  the  work  of  Dr.  Lee,  the 
model  confirmed  that  when  the  difference  between  the  mode 
goodness  or  quality  of  the  components  of  the  "best"  propo¬ 
sal  and  the  "second  best"  proposal  was  large,  the  most 
significant  internal  parameter  which  affected  the  selec¬ 
tion  was  the  weight  applied  to  each  component  (coefficient 
set).  When  the  weighting  difference  was  large,  the  propo¬ 
sal  with  the  "best"  modal  quality  was  less  likely  to  be 
selected  than  when  weighting  differences  were  small. 

The  relative  effectiveness  of  numerical  scoring 
and  color  scoring  as  discriminators  was  substantially 
dependent  on  the  nature  of  the  relative  difference  in  the 
quality  of  proposals  being  compared.  For  some  differences 
in  quality  of  proposals,  color-scoring  provided  signifi¬ 
cantly  less  preference  for  the  "best"  proposal  them 
numerical  scoring  provided.  The  inconsistency  of  dis¬ 
crimination  provided  by  color  scoring  is  explained  by  the 
"broad  banding"  of  the  four- increment  color  score  scale. 
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Evaluations  of  two  proposals  which  fall  on  different 
sides  of  the  boundary  between  two  color  bands  will  be 
discriminated  by  color  scoring.  However,  if  the  evalua¬ 
tions  of  two  proposals  (which,  theoretically  may  differ 
by  as  much  as  40$)  fall  within  the  same  color  band,  the 
color  scoring  system  will  not  diff erentiate  between  them. 

The  positive  features  of  numerical  scoring  when 
compared  with  color-scoring  are: 

(1)  Absolute  weights  may  be  allocated  to 
attributes  before  evaluation  and  scoring. 

(2)  The  inclusion  of  adjustment  parameters  (B0) 
which  can  wash  out  or  bias  final  scores 

is  precluded. 

(3)  Small  differences  in  evaluations  are 
recognized  in  the  scores  allocated  to 
attributes  and  are  discriminators  of  the 
outcome . 

The  disadvantages  of  numerical  scoring  are: 

(1)  A  degree  of  precision  of  evaluation  is 
implied  which  is  not  always  realistic, 
particularly  when  dealing  with  the  concept¬ 
ual  attributes  of  proposals. 

(2)  Numerical  scores  imply  a  sense  of  absolute¬ 
ness  which  inhibits  the  exercise  of 


qualitative  judgment  by  the  SSA 


(3)  Evaluators  tend  to  be  reluctant  to  use  the 
full  range  of  scores,  clustering  results  into 
a  narrow  band,  so  reducing  the  discriminating 
power  of  the  process. 

(4)  It  is  sometimes  difficult  to  obtain  agreement 
on  relative  weights. 

(5)  Extreme  responses  are  not  highlighted  (e.g., 
non-conformance ) . 

In  comparison,  color  scoring  offers  the  following 
advantage  s : 

(1)  A  convenient  and  powerful  means  by  which  a 
comparative  overview  of  the  quality  of  com¬ 
peting  proposals  may  be  visualized  is  provided. 

(2)  Subjective  values  of  attributes  may  be  scored 
with  high  levels  of  agreement, 

(3)  Extreme  responses  are  highlighted. 

(4)  The  SSA  is  provided  with  considerable  scope 
for  qualitative  judgment. 

The  disadvantages  of  color-scoring  are: 

(1)  Significant  differences  in  objective  evalua¬ 
tions  of  attributes  may  not  be  recognized  in 
the  scoring  process  as  discriminating  factors. 

(2)  Attributes  cannot  be  objectively  weighted  to 
highlight  comparative  importance. 
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(3)  The  process  permits  the  washing  out  or 

biasing  of  evaluation  results  by  the  intro¬ 
duction  of  an  adjustment  parameter  (B0). 

Both  methods  of  scoring  have  unique  advantages 
and  disadvantages.  Whether  one  method  or  the  other  is 
appropriate  is  dependent  on  the  nature  and  structure  of 
the  particular  source  selection  involved.  It  is  concluded 
from  this  study  that  the  choice  of  the  appropriate  method 
of  scoring  is  influenced  by: 

(1)  The  maturity  of  the  concept  being  considered. 

(2)  The  relative  importance  (weights)  of  the 
key  attributes  of  the  decision. 

(3)  The  resources  available  to  the  source 
selection  activity. 

(4)  The  management  style  of  the  SSA. 

Maturity  of  Concept 

The  maturity  of  the  concept  being  considered 
strongly  controls  the  level  at  which  a  proposal  may  be 
evaluated.  When  the  concept  is  novel  and  the  proposal  is, 
in  effect,  a  projection  of  what  might  be  done  based  on 
broad  assumptions,  then  the  evaluation  can  only  be  realist¬ 
ically  scored  at  a  qualitative  level.  Evaluating  human 
skills  such  as  expectations  of  management  or  innovative 
capabilities  is  also  highly  conceptual  and  only  able  to 
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be  satisfactorily  expressed  in  qualitative  terms.  Con¬ 
versely,  when  standard  and  predictable  techniques  and 
practices  of  mature  concepts  are  being  evaluated, 
quantitative  scoring  of  evaluations  can  be  done  with 
confidence  and  precision.  A  single  source  selection  may 
involve  a  mix  of  novel  and  mature  concepts.  For  example, 
a  technical  area  may  encompass  a  variety  of  well-developed 
concepts,  whereas  the  corresponding  logistics  area  may  be 
one  in  which  the  implications  of  the  systemic  application 
of  the  technology  is  entirely  novel. 

Weights  of  Attributes 

In  some  source  selections,  the  weight  of  the 
decision  may  rest  heavily  on  a  particular  attribute.  In 
others,  weights  of  attributes  may  be  about  equal.  Even 
at  lower  levels  of  evaluation  such  as  the  factor  level, 
it  may  be  necessary  to  weight  the  sub-factors  to  prevent 
the  important  attributes  from  being  swamped  by  the  many 
trivial  attributes.  Numerical  scoring  methods  allow  the 
use  of  definitive  weights  when  needed.  Color  scoring  is 
weak  in  its  ability  to  reflect  weightings  but  has  the 
power  to  highlight  component  deficiencies  when  weighting 
is  not  important. 


Source  Selection  Resources 

The  major  resources  available  to  a  source  selection 
activity  are  personnel,  time  and  money.  Personnel  may  be 
limited  in  numbers  or  specific  skills.  Time  available 
may  limit  the  depth  of  evaluation.  Money  resources  may 
determine  the  extent  of  investigation  of  proposed  solutions 
or  restrict  the  amount  of  outside  assistance  that  can  be 
brought  to  bear  on  the  source  selection.  All  of  these 
resource  constraints  may  reduce  both  the  effort  that  can 
be  put  into  evaluating  the  attributes  of  each  proposal, 
and  the  precision  with  which  the  attributes  may  be  scored. 
As  the  potential  for  precision  of  evaluation  is  reduced, 
color  scoring  becomes  a  more  suitable  technique  than 
numerical  scoring. 

Management  Style  of  the  SSA 

Simon  has  written  that  management  and  decision¬ 
making  may  be  viewed  as  synonymous  ( 1 6 : 1 )  .  The  management 
style  of  the  SSA  is  an  important  consideration  in  selecting 
the  source  selection  structure.  The  structure  should 
provide  the  S£A  with  the  kind  of  information  he  needs  to 
be  able  to  make  an  effective  decision  within  his  own  frame 
of  reference.  Keen  and  Morton  (7:62)  classify  decision 
style  into  five  main  groups: 


.  rational  -  based  on  analytical  definition  of  all 
the  variables  to  obtain  the  best  decision. 

.  satisficing  -  based  on  effective  use  of  available 
information  to  obtain  an  acceptable  decision. 
.  procedural  -  based  on  following  through  standard 
organizational  procedures  toward  a  single 
outcome . 

.  political  -  based  on  the  use  of  personalized 

bargaining  between  organizational  units  to 
seek  an  acceptable  decision. 

.  individual  -  based  on  the  decision-maker ' s  own 
individual  assessment  of  the  information 
available  to  him. 

This  grouping  of  decision-making  styles  suggests 
that  different  decision-makers  will  seek  different  kinds  of 
information  on  which  to  act.  Rational  and  satisficing 
decision-makers  are  likely  to  feel  more  comfortable  with 
numerically-scored  information,  whenever  it  may  be  practic¬ 
ally  applied.  The  procedural  decision-maker  is  unlikely  to 
strongly  favor  either  numerical  or  color  scoring,  so  long 
as  he  is  satisfied  that  a  correct  procedure  has  been 
followed.  Political  and  individual  decision-makers  are 
more  likely  to  be  attracted  to  color  or  narrative  scoring 
techniques  as  being  compatible  with  their  own  styles  of 
management , 
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Choice  of  Scoring  Method 

The  wide  range  of  factors  bearing  on  the  effective¬ 
ness  of  a  particular  source  selection  process  suggests 
that  there  is  no  one  best  technique  for  scoring  proposals. 
The  requirement  that  "a  qualitative  rating  scale  will  be 
used  in  lieu  of  weighted  scoring  (l:9)"  unnecessarily 
inhibits  ASD  source  selection  personnel  from  exercising 
the  flexibility  to  choose  the  process  best  suited  to  each 
source  selection  situation. 

Within  a  source  selection  case,  different  areas 
may  merit  different  scoring  processes  according  to  the 
criteria  discussed  above.  The  color  scoring  system  does 
not  offer  sufficient  range  to  be  able  to  satisfactorily 
show  important  differences  in  many  areas  of  technical 
evaluation.  In  other  areas,  such  as  management,  color 
scoring  may  be  an  appropriate  tool  when  needed  to  indicate 
the  outcome  of  largely  subjective  judgments.  There  is  no 
overriding  reason  why  numerical  and  color  scoring  should 
not  be  separately  used  in  different  parts  of  the  same 
source  selection.  If  done,  it  would  present  the  SSA  with 
an  overview  of  both  the  objective  and  subjective  aspects 
of  the  total  evaluation.  Alternatively,  if  it  is  the 
preference  of  the  SSA,  scores  could  be  feasibly  converted 
to  an  "all  color"  or  "all  number"  presentation  at  the 
area  level. 
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When  values  can  be  determined  with  higher  precision 
than  afforded  by  a  four- increment  color  range,  numerical 
scoring  offers  greater  power  of  discrimination  of  the 
merits  of  proposals  them  that  offered  by  color  scoring. 

This  advemtage  should  not  be  foregone  in  those  groups  of 
attributes  for  which  numerical  scoring  is  appropriate. 

Other  perceived  problems  of  numerical  scoring  (clustering, 
agreement  on  weights,  and  extremes  not  highlighted)  may 
be  overcome  by  appropriate  techniques.  The  techniques 
described  by  Beard  (MAUM)  and  Waid  (comparison  matrices) 
provide  practical  and  convenient  ways  of  making  objective 
and  effective  weight  and  score  allocations  with  reliability 
and  repeatability.  Extreme  attribute  scores  (mainly  non¬ 
conformance)  may  be  simply  highlighted  in  the  evaluation 
presentation  or  treated  with  an  exclusion  rule  which 
eliminates  the  proposal  from  further  analytical  considera¬ 
tion. 

Color  scoring  is  a  suitable  technique  for  representing 
evaluations  of  subjective  and  highly  conceptual  attributes. 

It  recognizes  the  imprecision  inherent  in  such  areas,  yet 
presents  a  good  overall  comparative  picture  of  proposals. 
Extremes  are  highlighted.  Where  precision  of  evaluation 
is  possible,  color  scoring  tends  to  wash-out  significant 
differences.  It  presents  problems  in  allocating  relative 
weightings  when  weights  are  significant  to  the  decision, 
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and  allows  a  wide  variety  of  outcomes  because  of  the  scope 
for  implicit  adjustment  factors  in  the  process.  The  dis¬ 
advantages  of  color  scoring  can  be  minimized  by  management 
vigilance  and  skill  in  application,  together  with  careful 
consideration  of  the  suitability  of  areas  to  the  applica¬ 
tion  of  the  technique. 

Procedural  Aspects  of  Source  Selection 

The  prime  objective  of  the  source  selection  process 
is  to  obtain  an  impartial,  equitable,  and  comprehensive 
evaluation  of  competitive  proposals  which  will  result  in 
the  selection  of  a  source  which  will  offer  optimum  satis¬ 
faction  of  the  government's  requirements,  to  include  cost, 
schedule,  and  performance  The  wide  range  of 

conflicting  responses  obtained  from  interviewees  regarding 
the  ultimate  objective  of  the  process  tends  to  indicate 
that  personnel  involved  in  source  selection  fail  to 
approach  the  process  with  a  common  objective.  This  lack 
of  agreement  impacts  on  the  quality  of  the  final  decision 
and  reduces  the  overall  effectiveness  of  the  process.  The 
need  to  understand  and  work  toward  a  common  objective  in 
source  selection  cannot  be  overemphasized.  It  is  essential 
in  order  to  make  a  selection  based  on  that  objective. 
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Effectiveness  and  Efficiency 

During  the  evaluation  of  proposals,  source  sel¬ 
ection  personnel  are  confronted  with  vast  amounts  of 
data,  a  large  part  of  which  is  not  needed  to  make  an 
effective  decision  in  an  efficient  manner.  This  excess 
detracts  from  the  decision-maker's  main  tasks.  Provid¬ 
ing  source  selection  personnel  with  excessive  amounts 
of  data  inhibits  them  from  being  able  to  effectively 
identify  and  assess  the  small  amount  of  really  important 
information  needed  to  reach  a  decision  which  will  result 
in  satisfaction  of  the  government's  objectives. 

The  study  of  source  selection  cases  during  this 
research  found  examples  in  which  areas,  items  or  factors 
were  broken  down  into  many  attributes  for  evaluation. 

Often  there  was  high  multi-collinearity  between  some 
attributes,  indicating  that  they  did  not  contribute  to  the 
decision.  Some  respondents  to  interviews  expressed  concern 
at  the  proliferation  of  sub-division  of  evaluation. 
Evaluation  of  management  was  cited  as  a  particular  area 
of  proliferation.  There  appears  to  be  a  tendency,  if  an 
area  is  recognized  as  critical  to  the  decision,  to  expand 
the  sub-headings  under  which  it  is  evaluated.  There  is 
the  danger  in  this  approach  that  proliferation  of  parts 
merely  leads  to  an  averaging  of  scores  and  obscures  what 
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is  important.  The  discrimination  of  the  process  is 
improved  by  keeping  the  number  of  attributes  small  and  by 
applying  differential  weights  to  them  according  to  their 
importance.  Helman  and  Taylor  (6:90)  suggest  that  only 
three  items  (planning,  organizing,  and  controlling)  should 
be  evaluated  in  the  management  area  and  that  each  item 
be  broken  into  no  more  than  four  factors. 

A  need  to  develop  a  better  way  to  consider  cost 
and  performance  tradeoffs  is  suggested  from  interview 
responses.  The  present  philosophy  of  source  selection  is 
to  associate  a  cost  with  a  technical  proposal,  identify 
acceptable  proposals  within  a  competitive  cost  range, 
and  then  obtain  best  and  final  offers  (BAFOs).  Although 
in  theory  the  budget  should  not  be  a  constraint,  and  the 
contract  should  be  awarded  to  the  offeror  who  best  satis¬ 
fies  the  government’s  requirements,  in  practice,  budget 
restrictions  sometimes  prevent  the  selection  of  the  best 
offeror.  A  tendency  to  emphasize  cost  limitation  at  the 
expense  of  technical  feasibility  may  not  be  the  best 
decision. 

Use  of  Scoring  Techniques 

Personnel  involved  in  source  selection  held  a  wide 
range  of  opinions  on  the  effectiveness  and  appropriateness 
of  methods  of  scoring  proposals:  numbers,  colors,  or 
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narrative.  There  was  little  objective  understanding  of 
the  implications  of  choosing  one  method  over  another,  and 
choice  was  largely  a  matter  of  subjective  preference 
based  on  experience.  The  model  experimentation  conducted 
during  this  research  suggests  that  there  are  circumstances 
in  which  the  method  of  scoring  proposals  significantly 
affects  the  outcome. 

Contractor  Inquiries 

There  was  a  broad  focus  on  the  concentration  of 
use  of  CXs  to  clarify  and  justify  the  work  of  evaluators 
beyond  what  was  required  for  contract  def initization. 

Much  of  the  use  of  CXs  was  felt  by  many  interviewees  to 
be  a  device  to  protect  the  organization  from  future  pro¬ 
tests  by  unsuccessful  offerors.  Proliferation  of  CI3 
tends  to  extend  evaluation  time  and  can  lead  to  technical 
leveling  and  raising  low  cost  proposals  to  more  favorable 
evaluation  levels.  Cases  were  seen  where  initially  poorly- 
rated  proposals  were  rescored  to  acceptable  levels  as  the 
result  of  Cl  actions.  It  was  observed  that  the  introduc¬ 
tion  of  the  four-step  solicitation  process  (20: l) 
contributed  to  a  large  reduction  in  the  use  of  CIs. 

Clearly,  a  balance  is  required  between  sufficient 
use  of  CIs  to  provide  adequate  contract  def initization  and 
an  inefficient  excess  of  CIs.  Many  personnel  felt  that  a 
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proper  balance  was  not  being  achieved 


Problems  of  Source  Selection 

The  major  problems  confronting  the  conduct  of 
source  selection  lie  in  meeting  effectively  and  efficiently 
the  objectives  of  impartiality,  equitability  and  compre¬ 
hensiveness.  The  descriptive  model  of  source  selection 
developed  in  this  research  shows  it  to  be  a  highly  complex 
process.  The  goals  of  the  process  are  not  always  clearly 
perceived.  The  effect  which  the  techniques  used  have  on 
the  interaction  within  the  process  are  not  widely  under¬ 
stood  by  personnel  involved  in  the  activity.  Procedural 
guidance  tends  to  be  fragmented  and  is  not  clear  on  the 
suitability  and  applicability  of  the  techniques  available 
to  evaluators  and  decision  authorities.  The  transitory 
nature  of  source  selection  teams  precludes  the  development 
of  depth  of  experience  in  many  personnel  key  to  the  process. 

A  logical,  consistent,  and  disciplined  approach, 
tailored  to  requirements,  is  necessary  to  provide  a  com¬ 
plete  and  objective  analysis  with  minimum  resources.  The 
process  should  efficiently  communicate  to  the  SSA  a  clear, 
complete,  relevant,  and  objective  analysis  which  will 
provide  a  reliable  basis  for  the  source  selection  decision. 
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Recommendations 

This  study  points  to  some  possible  ways  in  which 
source  selection  may  be  made  more  effective  and  efficient. 
Many  of  the  problems  experienced  in  making  efficient  and 
effective  source  selections  lie  in  the  limited  experience 
and  understanding  of  the  process  by  many  of  the  personnel 
involved.  Working-level  expertise  in  source  selection  is 
limited  because  of  the  relatively  short  time  many  partici¬ 
pants  spend  in  the  process.  However,  their  actions  impact 
on  decisions  involving  very  large  expenditures  of  money  and 
long-term  operational  commitments  by  the  Air  Force. 

A  significant  contribution  to  improvement  of  the 
operation  of  the  process  would  be  to  introduce  short  train¬ 
ing  programs  for  personnel  entering  source  selections.  The 
training  should  be  directed  toward  developing: 

( 1 )  A  common  understanding  of  the  Air  Force 
objectives  of  source  selection. 

(2)  A  knowledge  of  the  procedural  framework 
of  source  selection. 

(3)  An  appreciation  of  the  scoring  and  weighting 
techniques  available,  their  relative  ad¬ 
vantages  and  disadvantages,  and  their  scope 
of  application. 

The  training  program  should  emphasize  the  principle 
of  essentiality  in  source  selection.  A  source  selection 
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should  be  concerned  with  what  is  essential  to  the  decision 


It  should  focus  on  collecting  and  evaluating  essential 
data.  Efficient  source  selection  plans  should  restrict 
the  use  of  evaluation  items  and  limit  the  factors  to  a  few 
significant  headings  which  will  facilitate  meaningful 
discrimination  between  proposals.  The  literature  review, 
findings,  and  discussions  in  this  study  provide  some 
insights  of  the  source  selection  process  which  provide  a 
basis  upon  which  a  suitable  training  program  could  be 
built. 

In  parallel  with  training  development,  a  review  of 
source  selection  procedures  is  advisable.  The  following 
changes  are  recommended: 

(1)  Avoid  directed  use  of  specific  scoring  or 
weighting  techniques. 

(2)  Encourage  source  selection  planning  tailored 
to  the  specific  characteristics  of  the 
acquisition. 

(3)  Facilitate  integration  of  RFP  development 
with  source  selection  planning. 

(4)  Provide  guidance  on  relating  cost  evaluation 
to  technical  evaluation. 

The  better  under standing  of  procedures  and  policy, 
and  the  objectivity  that  can  flow  from  such  measures  should 
result  in  smaller,  more  purposeful  source  selection  teams, 
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and  more  powerful  decision  support  mechanisms  for  the  SSA. 

The  model  developed  here  is  an  attempt  to  describe 
the  complexities  of  the  source  selection  decision  process 
in  ASD.  It  is  not  complete,  and  does  not  purport  to  be 
so.  However,  it  is  hoped  that  it  will  provide  a  useful 
basis  from  which  to  improve  understanding  of  the  process. 
There  is  scope  for  much  further  work,  some  of  which  is 
suggested  in  the  preceding  pages. 
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GUIDE  TO  INTERVIEWS  WITH  SOURCE  SELECTION  PRACTITIONERS 


Introduction 

Outline  the  topic  and  scope  of  the  study  to  the 
interviewee. 

Obtain  details  of  the  interviewee's  background  and 
experience  in  source  selection. 

Specific  Points  of  the  Discussion 

The  purpose  of  this  section  of  the  interview  is  to 
obtain  the  interviewee’s  perceptions  of: 

(1)  The  ultimate  objective  of  the  source  selection 
process  and  the  effectiveness  and  efficiency 
of  the  process  toward  achieving  the  objective. 

(2)  The  comparison  between  numerical  and  color 
scoring  methods  and  comments  on  their  relative 
merits.  The  significant  level  of  discrimination 
in  numerical  scoring  systems. 

(3)  The  significance  of  Contractor  Inquiries  (CIs) 
and  Deficiency  Reports  (DRs)  to  the  decision 
process.  Is  the  effort  commensurate  with  the 
usefulness  of  CIs  and  DRs? 

(4)  Changes  that  could  be  made  to  improve  the 
source  selection  process. 
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Closing  Discussion 

Invite  additional  comment  on  the  source  selection 
process  which  might  aid  the  research. 
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